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Abstract 


This  report  describes  progress  in  vision  and  navigation  for  outdoor  mobile  robots,  at  the  Carnegie 
Mellon  Robotics  Institute  during  1987.  This  researctvwas  primarily  sponsored  by  darpa  as  part  of  the 
Strategic  Computing  Initiative.  Portions  of  this  reseat  were  also  partially  supported  by  the  National 
Science  Foundation  and  Digital  Equipment  Corporation. 

^  /  f  ■  ,  t 

We  are  pursuing  a  broad  range  of  perception  research  for'^uiding  outdoor  autonomous  vehicles.  In 
1987  we  concentrated  on  five  areas: 

1 .  Road  following.  We  expanded  our  road  tracking  system  to  better  handle  shadows  and 
bright  sunlight.'  For  the  first  time,  we  added  capabilities  for  recognizing  intersections. 

2.  Range  data  interpretation.  Our  range  interpretation  work  has  expanded  from  processing  a 
single  frame,  to  combining  several  frames  of  data  into  a  terrain  map.”  Building  terrain 
maps  requires  findinji  features  in  each  frame  of  data,  and  matching  those  features  between 
frames  to  get  precise  data  regisfratidrv  We'aisonsed  range  data  for  recognizing  cars. 

3.  Expert  systems  for  image  interpretation.  We  explored  finding  roads  in  very  difficult  scenes, 

without  relying  on  strong  a  priori  road  color  or  shape  models.  The  expert  system  that 
resulted  from  our  research  is  very  competent,  and  breaks jiew  ground  in  expert  system 
design.  -  - 

4.  Car  recognition.  We  recognize  cars  in  color  images  by  a  hierarchy  of  grouping  image 
features,  and  predicting  where  to  look  for  other  image  features.  We  use  only  weak  2-D 
shape  constraints,  rather  than  searching  for  a  particular  3-D  pose. 

5^  Geometric  cairriera~calibration. '  Oiir  new  method  for  calibration  avoids  complex  non-linear 
optimizations  found  in  other  calibration  schemes..  We  have  variants  for  cameras  with  near- 
linear  lenses  and  for  cameras  with  significant  local  distortions. 

This  report  begins  with  an  introduction,  chronology,  and  lists  of  personnel  and  publications.  It  also 
includes  papers  describing  each  of  the  research  areas. 
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Introduction 

This  report  reviews  progress  at  Carnegie  Melton  from  January  15,  1987  to  January  14,  1988  on 
research  sponsored  by  the  Strategic  Computing  Initiative  of  darpa,  DoD,  through  arpa  Order  5351 ,  and 
monitored  by  the  US  Army  Engineer  Topographic  Laboratories  under  contract  DACA76-85-C-0003,  titied 
’Road  Following’  Portions  of  this  research  virere  also  partially  sponsored  by  the  National  Science 
Foundation  contract  DCR~8604199  and  by  ttie  Digital  Equipment  Corporation  External  Research 
Program.  This  report  consists  of  an  introduction  and  overview,  and  detailed  reports  on  specific  areas  of 
research. 

Previous  Work 

The  ’Road  Following’  contract  at  CMU  has  been  funded  by  oarpa  since  January  15, 1985.  Our  goal  is 
to  build  perception  and  navigation  capabilities  for  a  mobile  robot  operating  in  the  real  outdoor  world. 
Work  in  earlier  years  was  reported  in  ’CMU  Strategic  Computing  Vision  Project  Report:  1984  to  1985” 
and  ’1986  Year  End  Report  for  Road  Following  at  Carnegie  Melton’,  both  available  as  technical  reports 
from  the  Carnegie  Melton  Robotics  Institute.  Those  reports  are  summarized  here  to  provide  background 
for  this  year's  work. 

The  two  dominant  themes  in  our  research  have  been  the  centrality  of  perception,  and  the  necessity  of 
a  complete  system.  Perception  is  the  single  stumbling  block  to  capable  mobile  robots.  The  limits  of 
robots  are  not  usually  in  planning  or  in  architectures,  but  rather  in  seeing  and  modeling  the  world  around 
them.  Our  research  has  therefore  focused  on  perception,  including  color  video,  range  data,  and  sonar 
mapping.  Perception  algorithms  developed  in  isolation,  however,  usually  fail  when  used  in  a  real  system 
context.  We  have  always  built  and  debugged  our  perception  algorithms  using  a  real  outdoor  mobile 
robot.  Only  with  a  complete  system,  running  in  realistic  situations,  is  it  possible  to  find  and  solve  the 
problems  of  navigation  in  the  real  world. 

Specific  technical  achievements  have  included  the  following: 

Road  Following.  Our  earliest  work  used  a  single  monochrome  camera  to  find  the  edges  of  a  sidewalk 
on  the  CMU  campus.  We  soon  switched  the  bulk  of  our  work  to  a  narrow  bicycle  path  on  nearby  Flagstaff 
Hill,  and  started  to  use  color  cameras.  We  have  developed  a  variety  of  road  tracking  methods:  line 
trackers,  color  classifiers,  oriented  edge  matchers,  and  histogram-based  methods.  The  most  successful 
of  those  classifies  each  pixel  according  to  its  color  and  texture.  It  maintains  a  model  of  typical  road  and 
grass  colors  and  textures,  and  updates  the  models  each  image. 

Range  Data  Interpretation.  We  have  processed  range  information  from  a  sonar  ring,  from  stereo 
vision,  and  most  inportantly  from  our  ERIM  scanning  laser  rangefinder.  The  sonar,  stereo,  and  initial 
ERIM  work  showed  that  we  could  detect  and  avoid  discrete  obstacles  such  as  trees.  Later  work  used  the 
ERIM  data  to  build  terrain  maps,  calculating  elevatton,  slope,  toughness,  and  discontinuities  for  each 
terrain  patch. 

Syatama.  Our  first  systems  drove  the  Terregator  vehicle  along  the  sidewalk  network  on  our  canpus. 
The  most  sophisticated  of  those  uses  map  information  to  switch  betiween  a  fbrward-looking  camera  and  a 
camera  that  peered  at  an  angle  to  see  around  comers.  More  recently,  our  main  test  area  has  been  on  a 
narrow,  tree-Hned,  twisting,  path.  The  Naviab  robot  has  slowly  driven  along  that  path,  tracking  the  road 
and  stopping  tor  obstacles  in  its  way.  We  have  implemented  our  algorithms  on  conventional  computers. 
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such  as  Vaxes  and  Sun  workstations,  and  on  several  generations  of  the  Warp  supercomputer. 

Overview 

During  1987  we  extended  our  work  on  perception  for  road  following  and  mapping,  and  started  new 
efforts  in  expert  systems  for  road  finding,  in  expert  systems  for  car  recognition,  and  in  camera  calibration. 
The  road  following,  mapping,  and  calibration  modules  were  used  to  drive  our  Navlab  robot.  Each  of  the 
modules  ir.  described  briefly  below,  and  at  length  in  the  following  chapters  of  this  report 

Road  following.  In  1987  we  attacked  problems  of  limited  camera  dynamic  range  and  of  intersection 
recognition.  The  previous  vision  system  successfully  drove  the  Navlab  on  our  narrow,  twisting,  test  path 
on  Flagstaff  Hill.  Its  main  limitations  were  that  it  could  not  deal  adequately  with  scenes  containing  both 
very  bright  and  very  dark  areas,  and  that  it  used  a  single  road  model  and  could  not  property  handle 
intersections. 

The  dynamic  range  of  a  camera  is  not  sufficient  to  see  both  heavily  shadowed  areas  and  areas  in  full 
sunlight.  Therefore  in  our  1987  system  we  used  two  video  cameras,  one  with  its  iris  set  open  for  seeing 
into  the  shadows,  and  one  with  its  iris  dosed  for  seeing  sunny  areas.  The  two  images  were  combined 
pixel  by  pixel  in  a  preprocessing  phase  which  required  keeping  separate  color  statistics  for  shaded  and 
sunny  areas,  and  modifying  our  dassification  scheme  accordingly. 

Having  a  single  road  model  meant  that  our  system  relied  on  predsely  known  road  width,  and  that  it 
could  not  recognize  an  intersection.  We  overcame  those  limitations  in  two  ways.  One  approach  used  a 
map  and  the  vehide's  position  to  predict  the  shape  of  the  road  or  intersection  in  the  field  of  view.  If  the 
vehide's  location  was  poorly  known,  it  was  often  necessary  to  try  both  a  road  model  and  an  intersection 
model,  and  decide  which  model  matched  best.  Another  approach  eliminated  specific  models,  and  used 
generic  road  cues  such  as  straightness  of  edges,  edges  parallel  to  each  other,  connectivity  of  the  road 
region,  and  so  forth. 

During  1987  we  installed  a  Warp  machine  on  our  Navlab  testbed  vehide.  Our  first  Warp  vision  system 
used  a  prototype  Warp,  running  the  1986  algorithms,  to  increase  vehide  speed  by  a  factor  of  five.  Later, 
we  installed  a  General  Electric  production  version  of  the  Warp.  Rather  than  push  for  speed,  in  our  1987 
system  we  used  the  increased  processing  power  for  processing  two  images  (for  shady  and  sunny  areas), 
for  searching  for  intersections  as  well  as  roads,  and  for  processing  at  higher  resolutions  to  be  able  to  see 
intersections  at  a  distance. 

The  1987  color  vision  system  is  described  by  Crisman  and  Thorpe  in  the  report  "Color  Vision  for  Road 
Following.” 

ERIM  Intarpraiation  and  map  building.  There  were  three  major  advances  in  range  data  processing 
this  year;  map  building  by  active  registration,  object  recognition,  and  effident  Warp  implementation. 

Inc^ual  frames  of  range  data  can  be  processed  to  describe  the  local  terrain,  either  by  building  a 
depth  map  or  by  extracting  features  such  as  smooth  patches  or  edges.  The  maps  from  a  single  frame, 
though,  cover  only  a  limited  area,  may  contain  noisy  data,  and  have  "shadows”  of  unseen  area  behind  tall 
objects.  This  year's  research  developed  methods  to  combine  data  from  multiple  scans  to  fill  in  the 
shadows,  extend  the  coverage,  and  increase  map  precision.  The  difficulty  in  combining  data  is  getting 
precise  registration  between  frames  of  data  talton  at  different  vehicle  locations.  The  techniques  we 
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developed  match  extracted  features  from  successive  frames  to  get  precise  registration  from  frame  to 
frame.  If  there  are  many  features,  the  results  are  quKe  good.  If  there  are  few  or  no  features,  registration 
must  depend  on  the  vehicle's  dead  reckoning;  but  in  featureless  areas,  a  higher  degree  of  error  can  be 
tolerated. 

We  have  demonstrated  the  use  of  range  data  to  recognize  cars.  The  range  image  is  segmented,  using 
cues  from  the  reflectance  channel  to  keep  regions  from  growing  across  edges.  The  extracted  surfaces 
are  then  matched  with  a  generic  car  model.  The  model  contains  both  surfaces  and  constraints  between 
surfaces,  such  as  'The  roof  must  be  roughly  perpendicular  to  the  side.”  Matching  proceeds  by  finding 
candidate  surface  matches,  and  checking  their  consistency  with  other  surface  matches  by  constraint 
checking. 

Range  data  processing  can  be  expensive.  It  is  largely  floating  point  calculations,  and  requires  many 
steps  ranging  from  removal  of  spurious  data  to  surface  fitting.  We  have  implemented  our  range  analysis 
on  the  Warp  processor  on  the  Navlab.  We  have  used  two  systems,  one  with  a  single  Warp  shared 
between  color  vision  and  range  processing,  and  the  other  system  with  a  dedicated  warp  for  each  of  the 
two  perception  rrxxJules.  This  code  has  been  transferred  to  Martin  Marietta  and  used  in  their  1987  demo 

More  details  about  range  data  processing  are  in  the  attached  paper  *3-0  Vision  for  Outdoor  Navigation 
by  an  Autonomous  Vehide",  by  Martial  Hebert  and  Takeo  Kanade. 

Road  finding  by  expert  system.  In  very  bad  lighting  conditions,  or  in  cases  where  there  is  no  a  priori 
road  location  or  shape  model,  finding  a  road  in  an  image  can  be  very  difficult.  We  developed  an  expert 
system  road  finder  that  uses  a  wide  array  of  knowledge.  This  system,  written  in  OPS,  alternates  between 
model  fitting  and.  region  evaluation.  At  each  step,  it  assigns  tentative  labels  (road,  grass,  tree,  sky,  etc.) 
and  weights  to  the  regions  it  thinks  it  can  label  reliably.  It  then  fits  a  geometric  road  model  to  the  labeled 
regions.  The  road  model  helps  constrain  region  labeling,  and  region  labels  help  constrain  model  fitting. 
Region  labeling  uses  mostly  local  information,  ^ut  a  region  and  its  neighbors.  Model  fitting  uses  global 
information,  about  the  labels  and  locations  of  all  regions  in  the  image.  Solutions  tend  to  start  with  a  few 
confidant  regions,  then  grow  to  neighboring  regions,  and  finally  settle  on  a  complete  image  interpretation. 
If  there  is  an  intersection  in  the  scene,  the  model  fitting  module  will  detect  a  problem  with  the  best  fit  and 
will  propose  an  additional  road  model  to  explain  the  discrepancy.  This  road  finding  expert  system 
successfully  labeled  many  difficult  scenes,  with  rruch  less  geometric  constraint  than  used  in  road 
following.  It  contributes  not  only  to  road  finding  but  also  explores  a  new  paradigm  in  expert  systems, 
alternating  between  labeling  and  model  fitting. 

The  OPS-{>ased  road  finding  expert  system  is  further  described  by  Taka  Fujimori  and  Takeo  Kanade  in 
Knowledge-Based  Interpretation  of  Outdoor  Road  Scenes.* 

Car  recognition.  We  have  created  a  program  that  recognizes  cars  in  color  images  without  the  use  of 
a  strong  3-D  model.  The  program  detects  image  features  and  groups  of  image  features  (lines, 
trapezoids,  ellipses),  and  uses  approximate  2-0  constraints.  This  is  in  contrast  to  the  vast  majority  of 
programs  which  perform  recognition  based  on  image  edges,  and  which  require  a  precise  model  of  the 
geometry  of  the  lines  in  the  object.  The  Landmark  Acquisition  Subsystem  Improving  Exploration 
(LASSIE)  begins  by  extracting  edges,  and  linking  them  to  form  lines  with  the  Miwa  iine  finder,  developed 
at  CMU.  LASSIE  then  proceeds  through  a  seri^  of  grouping  steps,  looking  tor  parallel  lines,  parallel 
lines  plus  two  others  to  complete  a  trapezoid,  sets  of  trapezoids  that  share  common  edges,  and  so  forth. 
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it  then  hypothesizes  interpretations  of  the  groupings  as  windows  and  the  roof  of  a  car.  Given  a  trial 
interpretation,  it  predicts  where  to  search  for  wheels,  lines  from  other  parts  of  the  car,  and  other  features 
to  confirm  the  match. 

Kart  Kluge  and  Hideki  Kuga  describe  LASSIE  in  "Car  Recognition  for  the  CMU  Navlab." 

Geometric  eameni  calibration.  Cameras  are  notoriously  hard  to  calibrate.  First  of  all,  it  is  difficult  to 
measure  lens  and  sensor  distortions,  which  can  vary  from  skewing  and  pin-cushion  distortions,  to 
rotations  caused  by  misaligned  CCO  chip  mounting.  Beyond  distortions,  it  is  also  difficult  to  get  exact 
positions  of  the  "center  of  focus"  of  a  typical  camera  with  compound  lens.  Most  solutions  either  assume 
precise  measurements  that  are  not  possible,  or  depend  on  nonlinear  optimizations  that  are  unreliable  and 
do  not  always  converge  to  the  correct  solution. 

We  have  developed  a  reliable,  simple  method  for  camera  calibration.  By  taking  two  pictures  of  a  grid  of 
points,  it  is  possible  to  extract  all  the  calibration  parameters.  The  grid  itself  needs  to  be  accurately 
measured,  and  its  motion  between  the  two  pictures  needs  to  be  accurately  known,  but  all  other 
parameters  are  treated  as  unknowns.  Thus,  the  output  is  not  only  the  camera's  lens  parameters,  but  also 
its  location  in  space  and  its  orientation.  Two  different  methods  of  extracting  the  parameters  have  been 
derived,  a  linear  method  that  smooths  errors  in  near-linear  cameras,  and  a  local  method  that  is  best  for 
cameras  with  significant  distortions  across  their  field  of  view. 

Camera  calibration  is  described  by  Keith  Gremban,  Chuck  Thorpe,  and  Takeo  Kanade  in  the  report 
"Geometric  Camera  Calibration  Using  Systems  of  Linear  Equations." 

Chronology 

Mar:  ERIM;  System  runs  on  the  WARP. 

Apr:  WARP:  Prototype  installed  on  Navlab. 

Apr:  VISION:  Road  Following  running  on  WARP. 

Apr:  WARP:  WPE  interface  to  GIL  installed. 

May:  WARP/Navlab  Demo  runs  at  50  cm/sec. 

May:  VISION:  First  OPS  based  road  scene  interpreted.  v 

May:  SIDEWALK  II.V:  Parallel  execution  of  pipeline. 

Jun:  CODGER:  Performance  monitoring  tools. 

Jul:  SIDEWALK  III:  Automatically  switches  between  cameras. 

Jul:  SIDEWALK  III;  Speed  changes  for  perception. 

Jul:  VISION:  Ten  OPS  based  road  scenes  interpreted. 

Aug:  VISION:  Complete  calibration  model  developed. 

Aug:  VISION:  First  car  identified. 

Aug:  VISION:  Segmentation  of  open  &  dosed  iris  images. 

Aug:  NAVLAB;  Power  system  upgrade. 

Sep;  VISION:  Park  Intersection  recognition. 

Sep:  WARP;  GE  machine  installed  on  Navlab. 

Oct;  CODGER:  Runs  with  uncertainty  model. 

Oct:  VISION;  New  interpretation  using  hough  space. 
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Nov:  VISION:  Reduced  resolution  digitization  runs. 

Dec:  PARK  87:  New  path  planner. 

Dec:  PARK  87:  Map  revision 
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Staff:  Paul  Allen,  Mike  Blackwell,  Tom  Chen,  Jill  Crisman,  Ralph  Hyre,  Bala  Kumar,  Jim  Moody,  Tom 
Palmeri,  Jean-Christophe  Robert,  Eddie  Wyatt 

Visiting  scientists:  Yoshi  Goto,  Taka  Fujimori.  Keith  Gremban,  Hide  Kuga,  Taka  Obatake 
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Technical  Report  CMU-RI-TR-87-24,  Carnegie  Mellon  University, 

The  Robotics  Institute,  November,  1987. 

Y.  Goto  and  A.  Stentz. 

Mobile  Robot  Navigation:  The  CMU  System. 

IEEE  Expert  :44-54,  Winter,  1987. 
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RushfekJt,  W.H.  Girodet,  and  P.  Schweizer. 

A  Feasibility  Study  for  Long  Range  Autonomous  Undemvater  Vehicle. 
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pages  1-13.  University  of  New  Hampshire.  June.  1987. 

K.  Ikeuchi  and  T.  Kanade. 

Modeling  sensor  detectability  and  reliability  in  the  sensor  configuration  space. 

Technical  Report  CMU-CS-87-144,  Camegie^tollon  University,  Computer  Science  Department,  1987. 

Takeo  Kanade  (ed.). 

Three-Dimensional  Vision  Systems. 

Kluwer  Academic  Publishers,  Boston,  1987. 

Takeo  Kanade. 

Image  Understanding  Research  at  CMU. 

In  Proceedings  of  Image  Understanding  Workshop,  pages  32-40.  SAIC,  Los  Angeles.  California, 
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T.  Kanade  and  M.  Fuhrman. 

A  Noncontact  Optical  Proximity  Sensor  tor  Measuring  Surface  Shape. 

Three  Dimensional  Vision. 

Kluwer  Academic  Publishers.  Boston,  1987,  pages  1S1-194. 

Larry  Matthias  and  Takeo  Kanade. 

The  Cycle  of  Uncertainty  and  Constraint  in  Robot  Perception. 

To  appear  in  Robotics  Research  4 , 1987. 

Larry  Matthias  and  Steven  A.  Shafer. 

Error  Modeling  in  Stereo  Navigation. 

IEEE  Journal  on  Robotics  and  Automation  239-248,  June.  1987. 

Matthias,  R.  Szeliski,  and  T.  Kanade. 

Kalman  RIter-based  Algorithms  for  Estimating  Depth  from  image  Sequences. 

Technical  Report  CMU-CS-87-185,  Carnegie  Mellon  University,  Corr^Xiter  Science  Department, 
December,  1987. 

T.  Okada  and  T.  Kanade. 

A  Three-Wheeled  Self-Adjusting  Vehicle  in  a  Pipe,  FERRET-1. 

International  Journal  of  Robotics  Research  6(4):60-7S.  Winter,  1987. 

D.  Reece  and  S.  Shafer. 

An  Overview  of  the  PHAROS  Traffic  Simulator. 

In  Proc.  of  the  2nd  International  Conference  on  Road  Safety.  Groningen,  The  Netherlands,  September, 
1987. 

Charles  Thorpe,  Martial  Hebert,  Takeo  Kanade,  and  Steven  Shafer. 

Vision  and  Navigation  for  Carnegie  Mellon  Navlab. 

Annual  Review  of  Computer  Science. 

Annual  Reviews  Inc.,  California,  1987,  pages  521-556. 

C.  Thorpe  and  T.  Kanade. 

1986  Year  End  Report  for  Road  Following  at  Carnegie  Mellon. 

Technical  Report  CMU-RI-TR-87-1 1 ,  Carnegie  Mellon  University,  The  Robotics  Institute,  May,  1987. 
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Pittsburgh,  PA  15213 

April  4,  1988 


Abstract 

This  paper  discusses  the  research  conducted  at  CMU  in  several  topic  areas;  increasing  the 
dynamic  range  of  current  cameras  to  better  handle  outdoor  scenes,  detecting  roads  in  heavily 
shadowed  scenes  and  in  rapidly  changing  illumination,  and  identifying  intersections.  This 
research  is  necessary  to  add  new  capabilities  to  our  road  following  vision  system.  We  built 
a  new  system,  using  the  results  from  our  research,  that  could  identify  roads  and  intersections 
in  shad^  scenes.  We  also  transferred  some  of  our  algorithms  to  the  WARP,  an  experimental 
supercomputer  developed  at  CMU,  to  test  the  capabilities  of  current  parallel  machines. 


1  Introduction 

Our  long-term  objective  is  to  create  color  vision  systems  capable  of  finding  and  following  roads  under  all 
conditions.  Our  ambitions  include  modeling  and  perceiving  dirt  roads,  city  streets,  and  expressways;  handling 
days  with  sun  as  well  as  rain,  snow,  or  dark  of  stormy  night;  and  both  driving  on  familiar,  well-mapped  roads, 
or  exploring  new  areas  without  the  aid  of  a  map. 

In  our  navigation  system,  the  vision  module  is  just  one  of  many  modules.  The  vision  module  is  responsible 
for  detecting  roads  in  the  camera  images,  the  pilot  module  uses  the  road  position  to  plan  a  path  for  the  vehicle 
to  travel,  a  map  of  the  world  is  updated  by  the  map  update  module,  and  the  helm  module  is  responsible  for 
sending  driving  commands  to  the  vehicle.  This  paper  is  concerned  only  with  the  vision  module  and  does  not 
discuss  the  functions  of  the  rest  of  our  navigation  system. 

Our  previous  research  [1]  has  built  the  PARK  I  system  capable  of  driving  our  Navlab  robot  van,  shown 
in  Figure  1,  on  a  narrow,  twisting,  asphalt  path  under  a  variety  of  lighting  conditions,  with  no  a  priori  map 
knowledge  but  with  a  known  road  model.  During  the  past  year,  we  have  concentrated  on  new  capabilities  in 
four  areas: 

•  increasing  the  dynamic  range  capabilities  of  our  cameras  to  handle  lighting  extremes, 

•  decreasing  our  reliance  on  a  fixed  road  model,  to  allow  us  to  track  roads  that  change  and  curve, 

•  using  a  map  to  predict  intersection  locations  and  shapes,  thus  allowing  us  to  recognize  intersections  and 

successfully  navigate  a  road  network, 

•  using  the  CMU  Warp  supercomputer  to  increase  the  processing  speed. 

Section  2  of  this  paper  discusses  our  PARK  I  system,  including  both  a  description  of  processing  and 
a  discussion  of  the  points  we  decided  to  improve.  Section  3  presents  specific  vision  research  towards  our 
objeaives.  Several  topics  were  researched  in  isolation,  some  of  which  were  used  in  a  PARK  n  system.  In 
section  4,  we  describe  the  PARK  n  system  incoiporating  the  new  vision  algorithms,  and  its  performance.  This 
section  also  discusses  the  WARP  implementation  of  our  PARK  I  system,  and  the  final  section  lays  out  the  plan 
for  our  next  round  of  research  tasks. 

'This  research  is  sponsored  by  the  Strategic  Computing  Initiative  of  DARPA,  DoD,  through  ARPA  Order  S3S1,  and  monitored  by 
the  US  Army  Engineer  Topographic  Laboratories  under  contract  DACA76-8S-C-0003,  titled  'Road  Following." 


2  Previous  Work 

2.1  The  PARK  I  Demo  System 

The  PARK  I  demo  system  digitizes  images  and  tries  to  find  the  best  straight  road  in  the  image,  using  color 
classification  and  a  Hough  voting  scheme.  This  is  done  in  five  phases;  System  Interface,  Pre-processing, 
Labeling,  Interpretation,  and  Model  Update.  The  data  and  control  flow  between  the  phases  is  shown  in  Figure 
2.  The  interface  phase  tells  the  pre-processing  phase  when  to  digitize  an  image.  The  pre-processing  phase 
transforms  the  camera  data  into  reduced  resolution  color  and  texture  images.  These  images  are  used  by  the 
labeling  phase  to  produce  a  road  probability  image,  in  which  each  pixel  describes  how  well  the  color  and  texture 
of  that  image  pixel  match  known  colors  and  textures  of  toad  pixels.  This  probability  image  is  passed  to  the 
interpretation  phase  that  finds  the  best  straight  road  in  the  image.  The  road  location  is  back-projected  onto  the 
ground  plane  and  passed  to  the  navigation  system.  The  road  location  in  the  image  plane  is  given  to  the  model 
update  phase  where  the  vision  module’s  idea  of  road  and  grass  colors  are  updated.  Each  of  these  phases  is 
described  in  more  detail  below. 

System  interface.  The  interface  between  the  vision  module  and  the  navigation  system  defines  the  function  of 
the  vision  module.  The  navigation  system  tells  perception  where  the  vehicle  should  be  when  it  takes  the  next 
picture.  The  vision  system  continuously  polls  the  navigation  system  for  the  vehicle  position,  and  digitizes  an 
image  when  the  vehicle  is  closest  to  the  specified  location.  The  image  is  then  processed  to  determine  the  road 
location  in  the  image. 

The  required  output  of  the  vision  system  is  the  location  of  the  toad  on  the  ground  plane.  The  ground  is 
assumed  to  be  locally  flat,  with  the  vehicle  sitting  on  the  same  plane  as  the  road  in  the  image.  The  coordinate 
frame  of  the  vision  output  is  defined  by  the  vehicle  location  when  the  image  is  digitized  as  shown  in  Figure 
2.  The  road  is  assumed  to  be  straight  and  of  known  width.  It  can  therefore  be  represented  as  a  rectangle  on 
the  ground.  Furthermore,  this  rectangle  can  be  described  completely  by  its  center  line.  The  location  of  this 
rectangle  on  the  ground  plane  can  be  described  by  two  parameters,  (x,  0):  the  x  position  of  the  center  line  as 
it  crosses  the  x  axis,  and  the  angle  at  which  the  center  line  crosses  this  axis.  These  two  parameters  arc  the 
description  of  the  road  that  the  vision  module  returns  to  the  system. 

Pre-processing.  The  pre-processing  phase  transforms  the  camera  data  into  images  that  can  be  processed  by 
the  labeling  phase.  In  our  case,  labeling  cannot  process  the  full  size  images  from  the  camera  due  to  time 
constraints.  Therefore,  the  pre-processing  phase  first  reduces  the  color  images  from  (480  x  512)  to  (30  x  32) 
pixels.  This  is  done  in  a  series  of  averaging  reduction  stages  that  create  a  pyramid  of  reduced  resolution  color 
images. 

Preprocessing  also  calculates  a  texture  value  for  each  pixel  in  the  (30  X  32)  image,  to  give  the  labeling 
phase  another  cue  besides  color.  Texture  is  calculated  by  counting  edges  found  by  a  high-resolution  Robert’s 
operator,  normalized  by  local  image  intensity  and  large-scale  edges.  Normalizing  by  intensity  assures  that  the 
texture  operator  will  have  similar  responses  to  similar  textures,  regardless  of  the  illumination  of  the  textured 
region.  Normalization  by  large-scale  edges  reduces  the  response  of  the  texture  operator  to  shadow  edges.  Grassy 
regions  and  trees  tend  to  have  many  local  variations,  which  produce  many  local  edges,  which  in  turn  produce  a 
strong  response  in  the  texture  operator.  Road  surfaces  are  usually  more  evenly  colored,  and  evoke  less  response 
from  the  texture  operator. 

At  the  end  of  this  phase,  the  four  (30  x  32)  images  representing  red,  green,  blue,  and  texture,  are  passed 
to  the  labeling  phase. 

Labeling.  The  goal  of  labeling  is  to  label  each  pixel  as  either  road  or  grass,  and  to  record  a  confidence  that 
the  pixel  is  correctly  labeled.  We  compute  the  label  and  confidence  by  comparing  each  pixel’s  color  and  texture 
with  multiple  road  and  grass  appearance  models.  Since  suimy  and  shaded  road  have  very  different  colors,  we 
need  a  separate  color  model  for  each  illumination.  Moreover,  since  colors  change  gradually  as  the  illumination 
changes  from  suimy  to  dark  shadows,  we  have  extra  models  to  represent  these  intermediate  colors.  We  found 
that  the  system  ran  well  using  four  road  models  and  four  grass  models. 
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Labeling  uses  these  color  models  in  a  standard  pattern  recognition  technique.  At  each  pixel,  a  likelyhood 
is  calculated  for  each  model,  and  the  model  with  the  highest  likelyhood  is  selected.  The  probability  of  this 
model  is  recorded  in  the  road-probability  image.  Positive  probabilities  in  this  image  signify  that  one  of  the  road 
models  had  this  highest  probability,  while  negative  pixels  signify  one  of  the  grass  models.  The  road-probability 
image  is  used  by  the  interpretation  phase. 

Interpretation.  The  interpretation  phase  is  tesp,  sible  for  transforming  the  road  probabilities  into  a  (x,  0) 
interpretation.  This  ground  plane  interpretation  can  be  transformed  into  the  image  plane,  resulting  in  an  equiv¬ 
alent  set  of  parameters:  c,  the  column  position  where  the  road  centerline  crosses  the  horizon  row,  and  <^,  the 
angle  with  which  the  centerline  crosses  the  horizon  row.  These  two  image  plane  parameters,  (c,  ^),  are  used  in 
a  Hough  space  interpretation  scheme.  Each  toad  pixel  votes,  using  its  probability,  into  the  Hou^  accumulator, 
for  all  road  shapes  that  contain  that  toad  pixel.  Each  grass  pixel  votes,  using  its  negative  probability,  against 
all  road  shapes  that  contain  that  grass  pixel.  The  best  road  location  in  the  image  is  given  by  the  coordinates  of 
the  peak  of  the  Hough  space.  This  location  is  passed  directly  to  the  model  update  phase,  and  is  transformed  to 
the  ground  plane  and  sent  to  the  navigation  system  interface  for  path  planning  and  vehicle  guidance. 

Model  Update.  The  model  update  phase  is  responsible  for  adjusting  the  statistical  color  model  of  the  different 
road  and  grass  models.  Tliis  adjustment  is  necessary  when  the  illumination  conditions  ate  changing,  or  if  the 
toad  or  grass  colors  change.  The  texture  model  is  not  adjusted,  since  texture  remains  constant  regardless  of  the 
lighting  or  the  toad  or  grass  colors.  After  the  toad  is  found  by  the  labeling  phase,  we  know  exactly  which  pixels 
in  the  image  ate  toad  pixels  and  which  are  grass  pixels.  Although  we  know  the  toad/grass  type  of  each  pixel, 
we  use  only  those  road  and  grass  pixels  that  are  not  near  the  toad  edge  for  updating  the  model.  This  prevents 
the  corruption  of  the  color  model  due  to  errors  of  modeling  curved  toads  with  strai^t  road  models.  For  each 
of  the  road  and  grass  regions,  we  perform  a  nearest  mean  clustering  technique  for  all  of  the  data  in  the  road 
and  grass  regions.  This  technique  groups  color  data  into  a  set  of  models  where  the  color  data  in  each  individual 
model  is  described  by  color  means  and  covariances.  The  clustering  technique  selects  the  set  of  models  whose 
mean  values  are  the  most  separated,  and  whose  clusters  are  the  most  compact.  The  means  and  variances  of  the 
resulting  classes  ate  then  used  by  the  labeling  phase  to  label  the  next  image. 

22  Lessons  Learned  from  the  PARK  I  Demo  System 

Under  many  conditions,  the  PARK  I  system  was  rather  robust  On  days  when  the  light  was  fairly  constant  the 
vision  module  worked  perfectly  almost  all  of  the  time.  In  the  times  that  the  solution  was  imperfect,  it  was 
not  far  from  the  correct  solution,  so  that  the  Navlab  drove  on  the  edge  of  the  road  instead  of  down  the  center. 
However,  there  were  some  limitations  with  this  system  that  needed  to  be  addressed. 

Increased  Dynamic  Range.  The  need  for  an  increased  dynamic  range  in  the  camera  is  especially  apparent  on 
bright  simny  days,  when  the  vehicle  is  in  the  sunlight  and  the  vision  module  is  processing  images  containing 
dark  shadows.  In  these  dark  shadows,  the  pixels  have  such  low  values  that  the  road  and  grass  pixel  values 
are  indistinguishable.  Likewise,  when  the  vehicle  is  in  the  shade,  sunlit  regions  of  the  image  are  saturated, 
and  again,  road  and  grass  are  indistinguishable.  We  need  some  way  of  increasing  the  dynamic  range  of  the 
perception  system  to  handle  these  difficult  scenes. 

Changing  Illumination.  The  labeling  phase  depends  on  knowing  the  colors  of  toad  and  grass  from  the 
previous  image.  At  the  end  of  each  step,  we  determine  new  road  and  grass  colors,  so  that  the  system  can  adapt 
to  changing  conditions.  When  illumination  changes  gradually  or  when  the  system  slowly  enters  shadowed 
regions  from  sutmy  regions,  the  system  can  update  the  color  model  and  successfully  track  the  road.  In  this 
sense,  the  system  is  capable  of  adapting  to  the  envirorunent  However,  when  the  illumination  changes  rapidly, 
as  when  the  sun  goes  behind  a  cloud,  the  color  model  used  for  labeling  can  be  inaccurate,  and  the  toad  can  be 
tnisclassified.  Therefore,  we  want  reliable  methods  of  distinguishing  road  from  grass  tegitms  that  do  not  solely 
rely  on  color  models  derived  from  the  previous  image. 
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Intersection  Detection.  As  in  any  vision  system,  the  interpretation  phase  needs  a  model  for  the  system  to 
match  in  the  image.  In  the  PARK  I  system,  the  model  is  implicitly  coded  as  a  straight  road.  We  needed  to  add 
models  and  algorithms  to  interpret  intersections  and  curving  roads. 

3  Research  and  Algorithm  Development 

To  address  the  problems  discussed  above,  and  to  expand  our  road  following  capabilities,  we  researched  in  the 
following  topics  independently: 

•  As  an  attempt  to  increase  the  dynamic  range  of  the  cameras,  we  developed  a  two  camera  pre-processing 
phase. 

•  Unsupervised  learning  was  tried  for  dealing  with  rapidly  changing  lighting  conditions. 

•  Two  different  ideas  for  recognizing  intersections,  a  convolution  interpretation  scheme  and  a  region  search¬ 
ing  algorithm,  were  developed. 

The  algorithms  that  were  successfully  developed  and  that  met  speed  and  accuracy  requirements  were 
integrated  into  a  PARK  n  demo  system. 

3.1  Two  Cameras 

Since  the  dynamic  range  of  available  video  cameras  is  not  large  enough  for  bright  suimy  days  with  dark  shadows, 
we  are  trying  a  two  camera  system  with  fixed  iris  settings.  One  of  the  cameras  has  its  iris  open  to  peer  into 
the  shaded  areas  of  the  scene.  The  image  from  this  camera  is  called  the  bright  image.  The  other  camera  has  a 
closed  iris  which  is  used  for  looking  into  sunlit  areas.  The  closed  iris  image,  the  dark  image,  has  an  average 
intensity  less  than  the  open  iris  image.  By  combining  the  bright  and  dark  images,  we  can  indirectly  increase 
the  dynamic  range  of  the  sensor. 

To  test  this  idea,  we  replaced  the  pre-processing  phase  of  the  PARK  I  system  with  a  new  pre-processing 
phase  that  digitizes  two  color  images  and  produces  one  reduced  resolution  color  image  and  a  texture  image. 
These  images  receive  some  of  their  pixel  data  from  the  bright  image,  and  some  of  their  pixel  data  from  the 
dark  image.  This  combined  image  is  fed  into  the  labeling  phase. 

First  the  digitized  images  are  reduced  to  form  bri^t  and  dark  color  image  pyramids.  Next,  each  image 
pyramid  is  run  separately  through  the  texture  subroutine,  producing  a  bright  and  dark  texture  image.  Then  the 
two  color  images  are  combined,  and  the  two  texture  images  are  combined.  For  each  pixel  in  these  combined 
images,  we  select  a  value  either  from  the  dark  image  or  from  the  bright  image.  This  is  better  than  trying  to 
do  any  sort  of  pixel-level  averaging  or  combination  since  these  techniques  would  distort  color  values  of  the 
images.  So  the  combination  step  chooses,  for  each  pixel,  which  image  to  use,  and  gets  the  ted,  green,  blue, 
and  texture  values  from  that  image.  It  also  produces  a  mask  image  that  records  from  which  image,  bright  or 
dark,  each  pixel  was  seleaed. 

For  selecting  which  pixels  should  be  placed  in  the  combined  images,  we  use  a  simple  thresholding 
technique.  We  first  apply  a  threshold  to  each  pixel  in  the  dark  image.  If  its  pixel  value  is  less  than  the  threshold 
value,  then  the  data  in  the  dark  image  is  inadequate.  Therefore,  this  combined  pixel  is  copied  from  the  bright 
image.  If  the  dark  image  pixel  value  is  greater  than  the  threshold,  then  this  pixel  is  usable,  and  is  copied 
from  the  dark  image  into  the  combined  image.  A  mote  sophisticated  algorithm  could  perhaps  improve  system 
performance  by  dividing  the  image  into  surmy  and  shaded  pixels  instead  of  just  dark  and  bright  pixels.  However, 
separating  surmy  from  shaded  may  be  just  as  difficult  as  the  original  problem  of  separating  road  from  non-road. 

By  running  this  combination  step,  and  carefully  setting  the  threshold  parameters,  we  ate  able  to  succeed 
in  increasing  the  dynamic  range  of  cameras  to  handle  images  containing  dark  shadows  and  bright  sunlight. 
Unfortunately,  these  parameters  need  to  be  individually  set  for  the  conditions  on  each  day  that  we  run  the 
vehicle.  Therefore,  we  would  like  this  threshold  to  adapt  automatically.  Even  with  the  conea  threshold,  there 
are  problems  with  the  colors  changing  within  a  shadow  and  with  rapid  and  lai^e  illumination  changes,  such  as 
the  sun  going  behind  a  cloud.  Sunlit  areas  ate  fairiy  evenly  illuminated,  and  the  same  surface  type  produces 
approximately  the  same  image  colors.  But  shaded  areas  ate  illuminated  by  reflections  from  the  sky,  from 
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clouds,  and  from  surroundings  like  trees  and  buildings.  The  color  of  the  illumination,  and  thus  the  color  of 
the  image  pixels,  changes  noticeably  depending  on  the  color  of  the  reflecting  objects.  More  research,  including 
fundamental  work  in  optics  of  materials,  is  needed  to  be  able  to  properly  classify  road  and  nonroad  under  all 
illumination  conditions. 

32  Unsupervised  Learning  Labeling 

When  the  lighting  conditions  change  rapidly,  our  PARK  I  system  fails,  since  it  relies  on  color  models  from  the 
previous  image.  Because  illumination  can  change  rapidly  and  unpredictably  outdoors,  models  calculated  from 
a  previous  image  may  not  correspond  to  the  colors  in  the  next  image.  We  would  like  to  build  a  labeling  phase 
which  does  not  rely  on  color  models  from  the  previous  image. 

The  input  to  this  labeling  module  is  the  same  as  the  PARK  I  system,  except,  of  course,  there  is  no  input 
color  model.  The  output  should  be  a  a  set  of  regions  in  the  image,  a  collection  of  which  would  completely  and 
exclusively  cover  the  road  in  the  image.  Our  basic  approach  is  to  use  a  standard  clustering  algorithm  to  group 
regions  that  have  similar  colors  [2]. 

To  run  this  clustering  algorithm,  the  number  of  classes  in  the  image  must  be  pre-seleaed.  First  the  pixels 
of  the  image  are  divided  randomly  into  each  of  the  classes.  The  mean  color  value  is  computed  for  each  class. 
Next,  each  pixel  is  labeled  as  belonging  to  the  class  whose  mean  value  is  closest  to  its  pixel  value.  The  mean 
color  is  then  recomputed  for  each  class.  The  sequence  of  label  by  closest  mean  and  calculate  mean  is  repeated 
until  few  or  no  pixels  change  their  class.  In  the  first  labeling  step,  the  majority  of  pixels  change  their  class, 
however,  in  the  second  labeling  step,  only  a  minority  of  the  pixels  change.  In  successive  steps,  the  number  of 
pixels  that  change  classes  continues  to  decrease.  If  run  long  enough,  this  will  converge  until  no  pixels  change; 
in  practice,  three  iterations  is  adequate.  Now  all  of  the  adjacent  pixels,  having  the  same  class  label,  can  be 
grouped  into  regions  using  a  cotmeaed  components  algorithm. 

The  biggest  problem  with  this  procedure  is  that  the  number  of  regions  found  in  the  image  is  quite  large. 
Searching  for  a  road  shape  becomes  impractical  if  there  are  too  many  regions.  Simple  methods  for  reducing  the 
number  of  regions,  such  as  shrinking  and  growing  or  merging  small  regions,  either  still  left  too  many  regions 
or  distorted  the  region  contours  to  too  great  an  extent.  For  unsupervised  labeling  to  be  applicable  to  real  time 
road  following,  we  need  to  improve  both  the  labeling  and  the  searching  methods  (see  Sections  .4  below).  This 
segmentation  system  is  not  directly  applicable  in  our  PARK  I  system,  since  its  interpretation  phase  depends  on 
road/grass  labeled  pixels.  There  are  no  semantic  labels  associated  with  the  tmsupervised  learning  scheme.  We 
are  in  the  process  of  developing  interpretation  methods  for  this  scheme. 

33  Interpretation  Using  Convolution 

One  of  the  main  reasons  for  the  robusmess  of  the  PARK  I  system  was  the  road  model  fitting.  We  used  a  Hough 
transform  to  globally  calculate  most  likely  position  of  the  road,  regardless  of  local  misclassifications. 

We  want  to  retain  that  robusmess,  but  extend  the  class  of  shapes  from  just  straight  roads,  in  the  eariier 
system,  to  also  include  curved  roads  and  intersections.  For  these  more  complicated  shapes  we  replaced  Hough 
transforms  with  convolution,  using  a  template  mask  in  the  shape  of  the  predicted  road  or  intersection. 

Each  road  segment  or  intersection  is  modeled,  based  on  either  map  data  or  previous  scene  interpretations, 
as  a  set  of  polygons.  The  vehicle  planning  system  chooses  the  most  likely  set  of  roads  and  intersections  in  the 
field  of  view,  and  passes  those  shape  descriptions  to  the  vision  module.  The  vision  module  does  not  directly 
search  for  those  shapes  in  the  image.  As  the  distance  changes  between  the  vehicle  and  an  intersection,  for 
instance,  perspective  projection  will  change  not  only  the  location  of  the  intersection  in  the  image  but  also  its 
size  and  shape.  So  instead  the  pixels  from  the  image  are  projected  onto  the  ground,  and  the  search  for  the 
predicted  shape  is  carried  out  in  ground  coordinates. 

The  first  input  to  this  system  is  a  probability  image  which  contains  positive  probabilities  for  road,  and 
negative  probabilities  for  grass.  The  magnimdes  of  these  probabilities  correspond  to  the  confidence  that  that 
pixel  is  acmally  road  or  grass.  The  second  input  is  a  shape  description  of  the  expected  road  or  intersection  in 
the  scene.  The  output  of  this  system  is  the  most  likely  location  of  the  specified  road  shape. 
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First,  this  algorithm  back  projects  the  road  probability  image  onto  the  assumed  ground  plane.  Then  it 
convolves  a  mask  formed  from  the  description  of  the  road  or  intersection  with  the  ground  plane  probability 
image.  To  find  a  straight  road,  only  the  x  dimension  and  the  angle  of  the  road  need  be  found  to  describe 
the  position  of  the  road,  since  the  appearance  does  not  change  along  the  y  direction.  However,  to  locate  an 
intersection,  the  (x,  y,  9)  position  of  the  intersection  on  the  ground  plane  are  needed. 

To  test  this  idea,  we  replaced  the  interpretation  module  of  the  PARK  I  system.  We  provided  test  models 
for  the  roads  and  intersections  by  selecting  the  borders  of  the  road  in  our  test  images.  These  points  are  back- 
projected  onto  the  ground  plane,  giving  us  a  perfect  model  of  the  intersection  seen  in  the  image.  The  results  are 
very  good,  matching  the  road  in  almost  all  situations.  Matching  with  distorted  models,  or  with  models  picked 
from  a  different  view  of  the  same  scene,  gave  less  accurate  but  still  satisfactory  results. 

3.4  Region  Searching  Interpretation 

The  main  motivation  for  this  algorithm  is  to  apply  road  information,  such  as  edge  shape,  color,  road  shape,  and 
predicted  location,  to  help  decide  the  location  of  the  road  in  the  image.  This  scheme  frees  us  from  the  exact 
shape  description  that  is  required  by  the  convolution  interpretation  scheme  discussed  above.  In  real  navigational 
simations,  an  exaa  model,  in  general,  is  unknown.  This  algorithm  is  an  attempt  to  interpret  road  and  intersection 
scenes  using  general  constraints,  without  knowing  the  exact  shape  of  the  roads. 

The  input  to  this  algorithm  is  a  list  of  regions.  Each  region  includes  the  following  information;  road/grass 
label,  road  probability,  size,  neighboring  regions,  and  polygonal  approximations.  The  output  is  the  collection 
of  regions  that  make  the  best  road.  Best,  in  this  case,  is  evaluated  using  geometric  and  heuristic  information. 
We  search  various  combinations  of  regions  in  the  image  to  see  which  collection  forms  the  best  road.  Each 
collection  of  regions  is  called  the  candidate  road.  An  exhaustive  search  over  all  candidate  roads  would  be  much 
too  expensive,  so  we  use  a  Hill  Climbing  algorithm  to  constrain  the  search  space. 

This  algorithm  is  tested  by  replacing  the  interpretation  phase  of  the  PARK  I  system.  In  the  PARK  I 
system,  the  labeling  phase  labels  pixels  in  the  image.  This  algorithm  needs  these  pixels  to  be  grouped  into 
regions,  so  we  added  a  region  growing  step  to  the  labeling  phase  to  test  this  algorithm. 

Region  Growing.  Before  we  can  apply  the  region  searching  technique,  we  first  need  to  convert  the  classifi¬ 
cation  and  probability  images  into  a  list  of  regions.  This  is  done  in  the  following  steps: 

1.  Use  a  connected  components  algorithm  for  region  extraction. 

2.  Merge  each  small  region  into  its  most  similar  neighbor. 

3.  Approximate  the  remaining  regions  with  polygons,  retaining  neighbor  information. 

4.  Calculate  descriptors,  such  as  size  and  road/grass  probability,  for  each  region. 

Region  Searching  Algorithm.  Searching  starts  by  considering  the  initial  candidate  road,  Co  consisting  of  the 
regions  that  were  labeled  as  road  in  the  input  image.  We  first  evaluate  the  cost,  v  of  the  initial  candidate  Cq. 
The  algorithm  proceeds  by  either  adding  neighboring  grass  regions  to  the  candidate  road  or  deleting  toad  border 
regions  from  the  candidate  road.  A  neighboring  grass  region  touches  the  candidate  road,  and  the  road  border 
regions  touch  at  least  one  grass  region.  The  algorithm  can  be  described  as  follows: 

1.  Try  expanding  road  by  evaluating  the  costs  of  all  the  candidate  toads  that  can  be  formed  by  adding  one 
neighboring  grass  region  to  the  initial  candidate  toad,  C,-.  Remember  which  of  the  grass  regions  G,  when 
added  to  the  initial  candidate,  gave  the  lowest  cost  of  vq. 

2.  Likewise,  try  shrinking  road  by  evaluating  the  costs  of  all  of  the  candidate  roads  that  can  be  formed  by 
deleting  one  of  the  toad  border  regions  from  the  initial  candidate  toad,  Ci.  Remember  which  of  the  road 
regions  R,  when  deleted  from  the  initial  candidate,  gave  the  lowest  cost  of  vr. 

3.  If  VR  is  lower  than  uq  and  v,  then  removing  one  region  from  the  toad  improves  the  road  model  fit. 
Cj+i  =  Cf  —  R  and  u  -  vr.  Go  to  1. 

4.  If  VG  is  lower  than  vr  and  v,  then  adding  a  grass  region  to  the  toad  improves  the  model  fit  C,>i  =  C,  +  G 
and  V  =  VG.  Go  to  1. 


17 


5.  Else  V  is  lower  than  vr  and  uq,  switching  the  label  of  any  one  tegion  will  not  lower  the  cost,  so  exit. 
The  final  result  region  C/ina/  is  the  collection  of  regions  in  the  image  that  fomi  the  best  road. 

Evaluation  of  Candidate  Roads.  The  confidence  of  a  road  interpretation  can  be  determined  by  a  combination 
of  the  following  constraints.  Notice  that  each  constraint  tries  to  enforce  a  particular  attribute. 

•  The  confidence  measure  comes  from  color  classification  of  the  pixels,  and  their  resulting  road  or  grass 
probabilities.  It  prefers  regions  that  have  high  road  probabilities. 

•  The  edge  straightness  metric  prefers  candidate  roads  whose  edges  form  straight  lines. 

•  The  road  width  constraint  reinforces  candidate  roads  that  have  the  correct  width. 

•  The  parallelness  measure  supports  candidate  road  that  have  parallel  edges. 

•  The  prediction  constraint  prefers  candidate  toads  that  are  spatially  close  to  the  prediction. 

Any  deviation  from  the  ideal  for  each  of  these  features  adds  to  the  cost.  We  form  a  weighted  linear  combination 
of  these  costs  to  calculate  the  total  cost  of  the  candidate  road,  u.  The  candidate  road  that  is  selected  by  the 
tegion  searching  algorithm  will  have  the  best  compromise  between  all  of  these  constraints.  This  cost  function 
can  be  easily  expanded  if  additional  constraints  are  needed  and  available. 

Results.  This  approach  is  very  promising  and  powerful.  When  given  good  predictions,  it  successfully  classifies 
all  our  test  images,  except  the  single  worst  shaded  intersection  image.  Its  main  drawback,  and  the  reason  it 
not  incorporated  into  our  demo  system,  is  that  it  does  not  guarantee  real-time  performance.  In  clean  scenes, 
relatively  few  regions  are  produced  by  pixel  labeling  and  region  growing,  and  the  initial  candidate  road  is  nearly 
correct.  Then  the  search  proceeds  quickly,  goes  through  few  iterations  of  generating  and  evaluating  candidate 
roads,  and  is  quite  efficient.  In  mote  complex  situations,  such  as  dappled  sunlight  or  leaves  scattered  on  the 
road,  it  is  possible  to  produce  literally  hundreds  of  separate  regions.  Searching  over  all  those  regions  is  not 
very  efficient.  Merging  small  regions  into  their  larger  neighbors  helps  with  search  time,  but  at  the  expense  of 
distorting  tegion  boundaries  and  possibly  corrupting  the  solution.  This  approach  requires,  and  merits,  further 
work  before  it  is  practical  in  real-time  road  following,  and  to  enable  it  to  not  to  rely  on  good  predictions. 

4  Demo  Systems 

During  the  course  of  the  year,  we  built  two  demo  systems  to  test  new  capabilities  of  vision  and  the  rest  of  the 
Navlab  navigation  system.  The  first  system  was  a  reimplementation  of  the  PARK  I  system  using  the  WARP 
machine.  The  second  system  was  built  to  test  some  of  the  new  algorithms  discussed  above. 

4.1  Warp  Demo  System 

The  PARK  I  system  required  about  10  seconds  to  complete  one  vision  cycle  on  a  Sun  3/160.  This  is  much 
slower  than  we  would  eventually  like.  Therefore,  we  wanted  to  run  the  PARK  I  system  on  the  prototype  wire 
wrapped  WARP  machine,  to  get  an  idea  of  the  speed  we  could  get  from  the  current  supercomputer  technology. 
A  prototype  WARP  was  installed  on  the  Navlab  for  this  purpose  [3].  The  WARP  is  an  experimental  high  speed 
computer  consisting  of 

•  an  array  of  four  cell  processors  that  can  perform  parallel  computations, 

•  itn  interface  unit  to  transfer  data  to  and  from  the  cell  array, 

•  Oiree  68020  processors  to  perform  higher  level  data  processing  and  sequencing,  and 

•  a  host  computer  that  links  high  level  programs  to  the  WARP  machine. 

The  WARP  group  provided  software  support  to  make  this  demo  possible.  Programs  for  the  WARP  machine 
ate  specified  in  an  Ada-like  language  called  W2  [4]  [5].  The  W2  compiler  produces  code  that  runs  in  parallel 
on  the  cells  of  the  Warp.  The  warpxall()  interface  allows  a  C  program  running  on  a  Sun  host  to  call  W2 
functions  on  the  parallel  processing  cells,  and  automatically  handles  all  data  transfer  and  format  conversions. 
We  divided  the  vision  program  into  modules  that  were  consistent  with  the  function  of  the  WARP; 
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•  Roberts’  gradient  operator 

•  Texture  operator 

•  Color  and  texture  classification 

•  Statistics  collection 

•  Statistics  class  adjustment 

•  Hough  space  voting 

All  of  these  modules,  except  for  the  Hough  module,  were  implemented  by  dividing  the  input  images  into  column 
swatches.  Each  of  the  cells  of  the  WARP  array  was  responsible  for  one  part  of  the  image  data.  The  results  were 
then  merged  when  they  were  read  from  the  array.  For  the  Hough  space  module,  the  Hough  space  accumulator 
was  divided  among  the  cells  by  column  swatches.  The  input  image  was  passed  to  all  of  the  cells,  therefore, 
each  cell  would  look  at  each  pixel  in  the  input  image  and  vote  for  all  the  toad  locations  within  the  range  of 
angles  described  by  its  swatch  of  the  Hough  accumulator. 

Recoding  our  algorithms  for  the  Warp  succeeded,  and  even  on  code  that  was  not  designed  for  parallel 
processing  we  achieved  a  speedup  of  1.5,  from  10  seconds  per  image  to  4.  We  learned  several  practical  lessons; 

•  Designing  for  parallelism  in  the  first  place  would  have  saved  us  much  more  time.  The  Warp  is  much  better 
at  some  things,  such  as  tegular  computations,  and  not  so  fast  at  others,  such  as  conditional  branching. 
Efficient  use  of  the  WARP  requites  designing  algorithms  with  these  constraints  in  mind. 

•  Even  with  the  computing  power  of  the  Warp,  standard  coding  optimizations,  such  as  unrolling  loops  and 
folding  constants,  are  still  important  time  savers. 

•  The  prototype  Warp  did  not  support  constructs,  such  as  variable  loop  bounds,  so  we  had  to  pad  data  to 
maximum  sizes,  thereby  wasting  computation. 

The  production  version  of  the  Warp,  which  we  have  since  mounted  on  the  Navlab,  helps  solve  those  problems. 

4  J  Park  System  n  Vision  Module 

The  PARK  II  demo  system  tested  some  of  our  experimental  algorithms  and  tested  the  new  PC  WARP.  Specif¬ 
ically,  we  wanted  a  system  that  used  the  two  camera  pre-processing  phase  and  the  convolution  interpretation 
algorithm.  The  vision  system  would  then  be  capable  of  recognizing  intersections  in  even  strong  shadows. 

An  outline  of  this  system  can  be  seen  in  Figure  3.  This  system  is  based  on  the  PARK  I  system  with  some 
important  changes.  The  pre-processing  phase  is  replaced  with  the  two  camera  system  that  was  discussed  in 
section  3.1.  The  labeling  phase  of  the  PARK  I  system  then  processes  the  combined  images.  The  interpretation 
phase  of  this  system  is  the  convolution  interpretation  scheme  discussed  in  section  3.3.  The  location  of  the  toad  is 
then  reported  to  the  navigation  system  through  the  new  interface  phase,  which  is  modified  to  transfer  intersection 
descriptions.  The  polygonal  model  of  the  road  or  intersection  is  passed  to  the  model  update  phase,  which  updates 
the  bright  and  dark  color  models  used  for  classifying  the  next  image.  The  more  detailed  description  of  the 
modifications  to  each  of  the  modules  is  given  below. 

Calibration.  We  expected  to  need  calibration  between  the  dark  and  bright  cameras.  By  mounting  the  cameras 
as  close  to  each  other  as  possible,  and  by  careful  bore  sighting,  we  were  able  to  get  nearly  perfect  alignment. 
For  the  ranges  of  interest,  and  at  reduced  resolution,  typical  errors  between  the  two  cameras  were  less  than  a 
pbiel. 

Calibrating  the  geometry  between  a  camera  and  the  vehicle  became  more  important  for  this  system.  In  the 
PARK  I  system,  vision  only  reported  locations  of  straight  roads,  so  calibration  of  distances  along  the  road  was 
not  crucial.  In  the  PARK  II  system,  vision  had  to  report  intersection  locations  as  well,  which  required  careful 
calibration  of  all  camera  parameters.  We  developed  and  used  a  calibration  scheme  that  uses  two  different  views 
of  a  grid  of  spots  to  deduce  camera  geometry  [6]. 
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Resolution.  To  identify  intersections  at  reasonable  distances,  the  (30  x  32)  resolution  of  the  road  probability 
images  was  not  high  enough.  The  branches  of  the  intersection  were  typically  smoothed  over  by  the  image 
reduction.  Therefore,  we  increased  the  image  size  to  (60  x  64).  We  also  developed  a  subroutine  to  digitize 
from  both  of  the  cameras  almost  simultaneously  by  digitizing  at  reduced  resolution  into  separate  portions  of  the 
frame  buffer  on  successive  camera  frames.  Our  input  color  images  were  therefore  (240  x  256)  rather  than  (480 
X  512). 

Convolution  Interpretation.  The  convolution  algorithm  is  used  for  the  interpretation  phase  of  this  system. 
The  navigation  system  predicts  the  shape  of  the  road  or  intersection  in  the  held  of  view,  or  of  multiple  possible 
objects  if  vehicle  position  is  imprecisely  known.  Each  shape  is  matched  to  the  road  seen  in  the  image.  The 
object  with  the  best  matching  shape  is  returned  as  the  identified  object,  and  its  best  fit  position  in  the  image  is 
projected  onto  the  ground  plane  and  returned  to  the  navigation  system.  This  shape  description  is  also  passed  to 
the  model  update  phase. 

The  location  of  a  road  can  be  described  with  two  parameters,  namely  (x,  9).  To  describe  the  location  of  an 
intersection  we  need  three  parameters  (x,  y,  9).  Unfortunately,  when  the  dimension  of  the  convolution  increases, 
the  computation  increases  exponentially.  In  order  to  reduce  the  computation  time  for  intersections,  we  use 
an  on  board  gyrocompass  to  localize  the  9  dimension,  thus  reducing  the  intersection  convolution  back  to  two 
dimensions,  (x,  y).  The  navigation  system  reads  the  gyro  information,  properly  rotates  the  intersection  model 
before  passing  it  to  the  interface  phase.  The  vision  module  then  matches  the  image  data  to  the  intersection 
model,  and  the  properly  translated  model  is  returned  to  the  navigation  system. 

Results.  The  system  ran  successfully  through  the  three  intersections  on  our  test  course.  Each  of  the  intersec¬ 
tions  had  different  shapes,  and  two  of  the  intersections  were  in  shaded  areas  of  the  site.  The  first  intersection 
was  located  on  a  curved  section  of  the  path,  and  the  branch  exiting  on  the  left  was  comprised  of  a  diHerent 
material  and  was  narrower  than  the  two  other  branches.  The  second  intersection  was  Y  shaped,  located  at  the 
crest  of  a  hill.  The  third  intersection  was  a  T  intersection  where  the  approaching  branch,  the  upright  of  the  T, 
was  angled. 

Two  of  the  results  of  our  system  are  shown  in  Rgure  4,  one  showing  the  algorithm  for  heavily  shaded  road 
interpretation,  and  the  other  showing  intersection  detection.  Each  result  is  shown  in  a  cluster  of  four  images. 
The  top  two  are  the  bright  and  dark  images  from  the  cameras.  The  lower  left  is  the  combined  image.  The  result 
of  the  segmentation  is  shown  in  the  lower  right.  The  position  of  the  detected  road  or  intersection  is  drawn  on 
top  of  the  dark  image  in  the  upper  left. 

We  made  many  runs,  including  runs  in  a  variety  of  lighting  conditions  and  longer  runs  than  any  of  our 
previous  systems  made.  The  navigation  system  used  perceived  roads  not  only  for  navigation  but  also  for  map 
building,  and  used  intersections  as  landmarks.  The  final  maps  showed  all  of  the  bends  and  curves  in  our  test 
course,  and  correctly  located  the  intersections.  Along  with  our  successes,  we  also  discovered  areas  for  further 
work. 

•  Shape  descriptions  It  is  impractical  to  depend  on  exact  shape  descriptions  of  intersections.  Moreover, 
if  the  shape  descriptions  are  significantly  incorrect,  not  only  is  the  calculated  location  somewhat  off, 
but  the  statistics  update  phase  may  incorrectly  update  its  model  of  road  and  nonroad  colors,  leading  to 
misclassifications  in  later  scenes. 

•  Assumed  ground  plane  The  ground  plane  assumption  is  unusable  for  the  second  intersection  of  our  test 
site,  where  the  intersection  lies  on  the  top  of  a  hill.  The  perceived  shape  of  the  intersection  changes 
dramatically  as  the  vehicle  pitches  forward  at  the  top  of  the  hill. 

•  Gyro  problems  We  had  difficulty  using  the  gyro  to  orient  the  intersections.  This  was  the  first  system 
to  attempt  using  the  gyro  on  the  Navlab.  The  device  would  drift  in  orientation  to  the  point  that  dead 
reckoning  of  the  vehicle  location  provided  better  orientation  information  than  did  the  gyro.  Therefore, 
when  matching  the  intersection  models  to  the  intersections,  we  would  often  receive  models  incorrectly 
oriented  from  the  navigation  system.  Future  systems  will  have  to  have  better  heading  information,  or  will 
have  to  search  for  intersections  in  x,  y,  and  9. 
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5  Future  and  Continuing  Work 

Our  ciurent  research  focuses  on  five  areas:  increasing  camera  dynamic  range  to  an  even  greater  extent,  dealing 
with  shadow  fringes,  using  approximate  shape  models,  getting  improved  ground  plane  models  from  direct  3-D 
sensors,  and  improving  unsupervised  classification  labeling. 

Because  our  two  cameras  had  a  fixed  iris  setting,  large  changes  in  illumination  could  cause  problems. 
When  the  sun  went  behind  a  cloud,  in  some  instances  even  the  bright  image  was  not  bright  enough  to  see  into 
the  shadows.  We  are  investigating  computer  controlled  irises  and  computer  controlled  digitizer  gains  to  try  to 
expand  further  the  dynamic  range  of  the  vision  system. 

Shadow  fringes  in  conjunaion  with  changing  illumination  also  provide  problems.  The  pixels  that  are 
most  often  misclassified  are  those  at  the  fringes  of  shadows,  where  the  illumination  changes  from  bright  to 
dark.  These  mixed  pixels  may  not  belong  imambiguously  to  any  one  color  class.  In  images  where  there  is  a 
sharp  shadow  line,  this  causes  problems  in  only  a  few  pixels.  In  other  images,  however,  with  dappled  sunlight 
filtering  through  trees,  a  large  portion  of  the  image  is  composed  of  shadow  fringe,  and  many  pixels  may  be 
misclassified.  We  are  working  on  methods  to  detect  shadow  fringes  and  discard  those  imreliable  labels. 

In  the  current  system,  an  exact  intersection  model  is  required  by  the  interpretation  phase  of  the  vision 
system.  However,  exact  models  are  not  realistic  in  real  navigational  situations.  They  are  also  impractical  even 
if  the  exact  shape  is  measured.  Therefore  we  would  like  to  continue  our  research  into  the  region  searching 
interpretation  method.  A  parallel  effort  in  our  project  has  developed  an  expert  system  image  interpreter  that 
finds  roads  and  intersections  in  very  difficult  scenes  and  which  has  only  weak  shape  models  [7].  This  expert 
system,  written  in  OPS,  takes  up  to  a  half  hour  per  image.  If  we  can  extraa  the  most  powerful  heuristics 
from  this  system  and  combine  them  with  our  experiments  in  region-based  road  finding,  we  can  perhaps  build  a 
powerful  yet  still  not  too  slow  system. 

We  would  also  like  to  improve  our  ground  plane  assumption  in  the  vision  system.  In  order  to  construct  a 
ground  surface,  we  need  three-dimensional  information  about  the  ground  surface.  Another  effort  in  our  project 
is  using  a  scanning  laser  range  finder  to  build  3-D  terrain  maps.  We  are  looking  at  methods  for  merging  this 
3-D  data  with  our  color  images  and  using  the  combined  data  to  give  true  intersection  or  toad  shape  and  location. 

We  are  also  continuing  our  work  on  the  unsupervised  clascification  method.  This  labeling  algorithm  should 
be  able  to  handle  changing  illuminations.  The  remaining  problems  are  that  the  labeling  scheme  results  in  far  too 
many  regions  to  be  reasonably  processed  in  real-time.  We  hope  to  develop  some  better  interpretation  methods 
or  region  reduction  algorithms. 

6  Conclusions 

Our  road  following  algorithms  have  made  significant  improvements,  but  there  remains  much  work  before  we 
achieve  our  goal  of  reliable,  autonomous,  road  following.  Specifically  we  believe  the  following: 

1.  Using  two  cameras  with  different  apertures  helps  significantly  with  the  problem  of  small  dynamic  range. 
We  continue  to  work  on  problems  of  rapidly  changing  illumination,  and  on  changes  of  color  at  shadow 
fringes  and  within  shadows. 

2.  Convolving  an  intersection  template  mask  with  classified  image  pixels,  projected  onto  the  ground  plane, 
works  for  recognizing  intersections.  We  continue  to  research  methods  for  finding  intersections  without 
exact  shape  models. 

3.  Region  based  road  finding  methods  can  help  by  using  not  just  individual  pixel  probabilities  by  other 
geometry,  such  as  edge  straightness  an  parallel  edges.  Such  heuristics  are  powerful  but  time  consuming. 

4.  Unsupervised  classification  has  promise  for  producing  cleaner  regions  for  region  based  methods.  Since  it 
does  not  rely  on  matching  with  known  color  models,  unsupervised  clustering  correctly  models  the  colors 
in  each  image,  and  produces  cleaner  region  boundaries. 

5.  Other  non-color  faaors  are  important,  such  as  fusion  with  three-dimensional  data  to  get  accurate  ground 
shape  models  and  plarming  for  parallel  implementation. 
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Figure  2;  The  PARK  I  System  and  the  Vehicle  Coordinate  Frame 
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figure  3:  The  PARK  n  System 
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Abstract 

This  paper  reports  progress  in  range  image  analysis  for  autonomous  navigation  in  outdoor  environments.  The  goal  c/  our 
work  is  to  use  range  data  from  an  E/t/M  laser  range  finder  to  build  a  three-dimensional  description  of  the  environment.  We 
describe  techniques  for  building  both  low-level  description,  such  as  obstacle  maps  or  terrain  maps,  as  well  as  higher  level 
description  using  model-based  object  recognition.  We  have  integrated  these  techniques  in  the  NAVLne  system  I  ID}. 


1.  Introduction 

Research  in  robotics  has  recently  focused  on  the  field  of  autonomous  vehicles,  that  is  mobile  units  that  ean  navigate  under 
computer  control  based  upon  sensory  information.  Several  eoniponcnis  are  involved  in  the  design  of  such  a  system.  A  high  level 
cognitive  module  is  in  charge  of  making  decision  based  on  the  perceived  environment  and  the  mission  to  be  carried  out.  Sensory 
modules,  such  as  video  image  analysis  or  range  data  analysis,  transform  the  sensors'  output  into  a  compact  description  that  can  be 
used  by  the  decision-making  modules.  Low-level  eonuol  software  eonvciis  decisions  into  actions  performed  by  the  hardware. 

In  this  paper,  we  focus  on  one  type  of  sensory  component,  the  analysis  of  range  data  for  an  autonomous  vehicle  navigating  in 
an  environment  with  features  such  as  trees,  uneven  terrain,  and  man-made  objects.  In  particular,  we  study  the  use  of  the 
Environmental  Research  Institute  of  Michigan  (ERIM)  laser  range  finder  to  perform  four  tasks:  obstacle  detection,  surface 
description,  terrain  map  building,  and  object  recognition  (Fig.  1-1).  Obstacle  detection  is  the  minimum  capability  required  by  an 
autorwmous  vehicle  in  order  to  navigate  safely.  Surface  description  is  needed  when  the  obstacle  detection  is  not  sufGcient  for 
safe  navigation,  in  the  case  of  uneven  terrain  for  example,  or  when  a  more  accurate  description  of  the  enviroiunent  is  needed,  e.g. 
for  object  recognition.  Terrain  map  building  is  the  process  by  which  surface  descriptions  from  different  vantage  poirus  are 
accumulated  in  a  consistent  map.  Object  recognition  capabilities  arc  required  when  the  vehicle  must  recognize  and  locate  knovm 
landmarks,  e.g.  a  traffic  sign,  in  order  to  carry  out  its  mission. 


Thil  totettch  wtt  tpocuoted  in  ptci  by  dakfa  mumu>tcU  by  Uic  U.S.  Amy  Un^inccrTopognphic  Lab  under  contna  DACATb^SS-C-COOS,  by  oawa  monitarad  by 
the  Air  Force  Avionic*  Laboratory  under  coniract  1*3301  S-KT-C*  1499,  and  by  nsi>  under  cuntraci  DCR*S604199. 
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Figure  1>1:  Range  data  processing 


2.  Intermediate  representations  of  range  data 


2.1.  Sensor  data 

The  ERIM  sensor  derives  the  range  at  each  point  by  measuring  the  difTcrence  of  phase  between  a  modulated  laser  beam  and 
reflection  from  the  scene.  A  two-minors  scanning  mechanism  directs  the  beam  onto  the  scene  so  that  an  image  of  the  scoie  is 
produced.  In  the  erim-alv  version,  the  field  of  view  is  ±40  in  the  horizontal  plane  and  30  in  the  vertical  plane,  flom  IS  to 
4S  .  The  resulting  range  image  is  a  64  x  2S6  8-bit  image.  The  frame  rate  is  currently  two  images  per  second.  The  nommal 
range  noise  is  0.4  feet  at  50  feec  The  sensor  also  produces  reflectance  images  in  which  the  value  of  each  pixel  is  the  amount  of 
light  reflected  by  the  target  Figure  2-1  shows  an  example  of  a  range  image  and  the  corresponding  reflectance  image. 

The  ERIM  sensor  presents  some  limitations:  Since  only  the  phase  difference  is  measured,  the  range  values  are  known  only 
modulo  64  feet.  This  causes  ambiguity  in  the  range  data.  The  resolution  degrades  rapidly  as  the  range  increases.  This  is  due  to 
the  divergence  of  the  beam,  which  produces  larger  footprints  os  the  distance  increases,  and  to  the  scanning  mechanism.  Since  the 
scanning  angles  are  discretized  using  constant  increments,  the  density  of  points  decreases  as  the  range  increases.  Points  may  be 
incorrectly  measured  at  the  edges  of  objects  due  to  multiple  reflections  of  the  beam.  This  effea  is  common  to  all  active  scanning 
techniques  and  is  known  as  the  ''mixed  points"  problem.  Wc  have  found  that  applying  a  median  Alter  to  the  original  image 
eliminates  most  mixed  points. 


(a)  Range  image:  the  darker  pixels  arc  closest 


(b)  Reflectance  image. 


Figure  2-1;  An  example  of  range  and  reflectance  images 
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2^  Vehicle  centered  coordinates 

The  rmw  dau  from  the  ERIM  scanner  represent  range  as  a  function  of  angles  0  and  $  of  the  two  minors. 

As  shown  in  Figure  2-2,  we  assume  that  a  coordinate  frame  if .  7^.  9  is  attached  to  the  scanner.  We  use  another  coordinate 
frame,  the  "vehicle"  frame,  V,J*,  K  to  express  the  measured  points  so  that  the  resulting  values  are  vehicle-centered  and  are 
therefore  independent  of  the  sensor  configuration.  We  can  thus  derive  the  coordinates  (x;y,z)  of  the  point  measured  at  pixel 
(row,coO  with  range  D.  If  ^  is  the  angle  between  (if,^)  and  the  direction  of  the  measured  beam  W  at  pixel  i,y,  8  is  the  angle 
between  (if,  i9)  and  the  direction  of  the  measured  beam  7  at  pixel  t,y,  is  the  starting  vertical  scanning  angle,  A  6  and  A  0  are  the 
angular  increments,  and  T  is  the  tilt  angle  of  the  scanner,  that  is  the  angle  between  the  planes  (H  J*)  and  (if,  as  shown  in  Figure 
2-2,  then  the  conversion  is: 

0»O  -  128)xA0 
-  ix  A0 

xsD(cos  0(cos  'icos  $  -sin  tsin  $)) 

3taOsin0 

z^Dicos  0(sin  tcos  ^  +cos  tsin  <i)) 


0. 


Figure  2-3  shows  the  data  of  the  image  of  Figure  2-1  after  conversion  to  vehicle  coordinates  as  viewed  in  the  direction  of  the 


HEASURCD  POINT 


2J.  Bucket  map 

hi  outdoor  environments,  the  ground  plane  (r,/*)  has  a  privileged  role:  the  terrain  can  be  modeled  as  a  funedon  z*/fx,y),  x 
and  y  being  the  coordinates  on  the  ground  plane.  In  order  to  take  advantage  of  the  ground  plane,  we  used  a  intermediate 
representauon,  the  bucket  map. 

A  bucket  map  is  defined  by  a  regular  grid  on  a  reference  horizontal  plane.  Each  cell  of  the  grid,  or  bucket,  contairts  a  set  of 
measured  points  as  shown  on  Figure  2-4.  The  points  within  a  bucket  may  all  be  from  the  same  image  or  from  several  consecutive 
images.  The  size  of  a  bucket  is  typically  30  ems  x  30  cms. 


Figure  2-3:  Overhead  view  of  the  data  of  Figure  2-1 


Figure  2-4:  The  bucket  map  scruemre 


3.  Obstacle  detection 

The  first  task  of  range  data  analysis  for  navigation  is  to  report  the  portions  of  the  enviroiunent  that  are  potentially  hazardous. 
We  must  identify  individual  objects  in  the  environment  that  the  vehicle  must  avoid.  Most  obstacle  detection  algorithms  combine 
surface  discontinuidcs  and  surface  slope  [6  11]  to  extract  untraversable  regions  in  the  image  using  a  vehicle  model  [3].  Faster 
algorithms  use  apriori  knowledge  of  the  tetrain,  e.g.  flat  groutxl  assumption,  by  computing  the  difference  between  the  range 
image  and  the  expected  ideal  image  [4].  Since  we  want  to  be  able  to  navigate  in  a  variety  of  environments,  we  chose  the  first 
approach  which,  although  computationally  expensive,  allows  us  to  handle  uneven  terrains.  Specifically,  we  identify  points  in  the 
bucket  map  at  which  the  elevation  exhibits  a  large  discontinuity  and  pwints  at  which  the  surface  slope  is  above  a  given  threshold. 
The  First  set  of  points  corresponds  to  the  edges  of  the  objects,  the  second  lies  within  the  surface  of  an  object  facing  the  sensor. 
The  obstacle  detection  algorithm  is  divided  into  four  steps: 

1.  Detect  elevation  discontinuities  on  the  bucket  map.  Tlic  discontinuities  are  computed  in  2x2  windows  around  each 
point; 

2.  Compute  the  surface  normal  for  each  bucket; 

3.  Detect  the  buckets  for  which  the  surface  normal  whose  angle  with  the  vertical  direction  is  greater  than  a  given 
threshold; 

4.  Extract  connected  regions  from  the  ret  of  buckets  detected  at  step  3  so  that  each  region  corresponds  to  an  object. 

Two  buckets  arc  coimecUKl  if  they  do  not  cross  tlic  line  of  elevation  discontinuity. 

The  rationale  for  using  two  criteria,  elevation  and  surface  normals,  is  that  the  surface  normals  are  meaningless  at  the  edge  of 
an  object,  and  the  elevation  cannot  be  used  alone  without  some  knowledge  of  a  ground-plane  on  which  objects  are  known  to  rest. 
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One  possible  undesirable  result  of  the  detection  algorithm  is  that  a  small  portion  of  the  terrain  might  be  reported  as  an  obstacle 
due  to  the  resolution  of  the  bucket  map.  This  error  can  be  corrected  only  when  a  more  complete  object  descripdon  is  created.  It 
does  not,  however,  significantly  affect  the  behavior  of  the  vehicle  since  only  small  regions  are  involved. 

For  the  purpose  of  obstacle  avoidance,  the  detected  objects  are  represented  by  polygons  on  the  ground  surface  in  order  to  be 
used  by  a  path  planner.  Figure  3-1  shows  a  range  image,  the  location  of  the  buckets  classified  as  parts  of  ol^ects,  and  the 
polygonal  representation  of  the  obstacle  map.  The  squares  indicate  the  buckets  in  which  the  objects  have  been  found.  The  large 
polygon  enclosing  the  map  is  the  boundary  of  the  portion  of  the  environment  seen  by  the  sensor. 


Figure  3-1:  Obstacle  Detection 


4.  Surface  description 

The  obstacle  detection  algorithm  is  sufficient  for  vehicle  navigation  in  a  simple  environment  which  includes  only  a  smooth 
terrain  and  discrete  obstacles.  A  typical  example  of  such  an  environment  occurs  in  road  following  applications.  We  need  a  more 
sophisticated  representation  in  two  cases: 

•  The  surrounding  terrain  is  uneven.  In  that  case,  part  of  the  envirorunent  that  may  be  hazardous  or  eostly  to  tuvigate 
catuiot  be  described  as  discrete  objects. 

•  The  mission  requires  the  recognition  of  specific  objects  given  apriori  models.  In  that  case,  the  mere  knowledge  of  the 
existence  and  position  of  an  object  in  the  world  is  not  sufficient,  we  need  a  more  detailed  description  of  its  shape. 

We  describe  surfaces  by  a  set  of  connected  surface  patches.  Each  patch  corresponds  to  a  smooth  portion  of  the  surface  and  is 
approximated  by  a  parameterized  surface.  In  addition  to  the  parameters  and  the  neighbors,  each  region  has  two  uncertainty 
factors:  a^,  and  Oj.  is  the  variance  of  the  angle  between  the  measured  surface  normal  and  the  surface  normal  of  the 

approximating  surface  at  each  point,  is  the  variance  of  the  distance  between  the  measured  points  and  the  approximating 

surface.  Those  two  attributes  are  used  in  the  object  recognition  algorithm. 

The  surface  description  is  obtained  by  segmenting  the  range  image  into  regions.  Several  schemes  for  range  image 
segmentation  have  been  proposed  in  previous  work  [  1  ].  These  techniques  are  based  either  on  clustering  in  some  parameter  space, 
or  region  growing  using  smoothness  criteria  of  the  surface.  We  chose  to  combine  both  approaches  into  a  single  segmetuation 
algorithm.  The  algorithm  first  attempts  to  find  groups  of  points  that  belong  to  the  same  surface,  and  then  uses  these  groups  as 
seeds  for  region  growing,  so  that  each  group  is  expanded  into  a  smooth  connected  surface  patch.  The  smoothness  of  a  patch  is 
evaluated  by  fitting  a  surface,  plane  or  quadric,  in  the  least-squares  sense. 

The  strategy  for  expanding  a  region  is  to  merge  the  best  point  at  the  boundary  at  each  step.  This  strategy  guarantees  a  near 
optimal  segmentation.  It  has,  however,  two  major  drawbacks  however  it  may  be  computationally  expensive,  and  it  may  lead  to 
errors  due  to  sensor  errors  on  isolated  points,  such  os  mixed  points.  To  alleviate  these  problems,  we  use  a  multi-resolution 
approach.  We  first  apply  the  segmentation  to  a  reduced  image  in  which  each  pixel  corresponds  to  a  n  x  n  window  in  the  original 
image,  n  being  the  reduction  factor.  This  first,  low-resolution,  step  produces  a  conservative  description  of  the  image  (Fig.  4-l.c). 
The  low-resolution  regions  are  then  expanded  using  the  full-resolution  image  (Fig.  4-l.d).  No  new  regions  are  created  at  full 
resolution.  Figure  4-2  shows  the  segmentation  of  an  image  of  uneven  terrain. 
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(a)  Range  image. 
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(b)  Edges  from  reflectance  image. 
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(c)  Low-resolution  segmentation  (n  =>  2). 
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(d)  Final  segmentation. 

Figure  4-1:  Range  image  segmentation 

In  addition  to  the  pure  region  segmentation,  we  use  the  edges  extracted  from  the  reflectance  image  to  improve  the  descripdon. 
In  the  low-resolution  segmentation  step,  pixels  that  coircspond  to  a  window  that  contains  at  least  one  edge  pixels  are  removed.  In 
the  full-resolution  step,  regions  are  expanded  so  that  they  do  not  cross  an  edge.  As  a  result,  edge  pixels  are  all  part  of  the  regions 
boundaries.  Explicitly  including  edges  improves  the  segmentation  in  two  ways:  First,  edges  that  correspond  to  low-amplitude 
occluding  edges  separates  regions  that  may  be  merged  in  the  range  image  segmentation.  Second,  reflectance  edges  can  delineate 
surface  markings  that  are  not  visible  in  the  range  image.  Figure  4-l.b  shows  an  edge  image  obtained  by  applying  a  10  x  10 
Catmy  edge  detector. 


5.  Terrain  map  building 

Map  building  is  the  process  of  combining  observations  of  an  unknown  terrain  into  a  coherent  representation.  Building  a  terrain 
map  serves  two  purposes:  It  allows  to  report  a  set  of  observations  as  a  product  of  an  exploration  mission,  and  it  improves  the 
performance  of  an  autonomous  vehicle  when  the  vehicle  traverses  a  previously  mapped  region.  Two  types  of  information  are 
kept  in  a  terrain  map:  the  low-lcvcl  measurements  that  arc  accumulated  in  a  bucket  map  as  described  in  Section  23,  and  the 
terrain  or  ot^ect  features. 

The  main  issue  in  the  map  building  process  is  the  matching  and  enhancement  of  a  map  built  from  measurements  acquired  from 
different  vantage  points.  This  issue  presents  some  challenging  aspects.  Since  we  do  not  put  any  constraints  on  the  vehicle's 


(a)  Range  image. 


(b)  Edges  frem  leilectance  image. 
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(d)  Final  segmentation. 

Figure  4-2:  Range  image  segmentation 


trajectories,  observations  of  the  environment  may  be  radically  different  from  one  observation  to  the  other.  This  requires  the 
idemificaiion  of  common  features  between  observations,  as  well  as  the  use  of  uncertainly  on  the  various  vehicle  position 
estimates.  The  insertion  of  a  new  frame  in  the  map  must  (licrcfore  proceed  in  three  steps;  Hrst,  the  current  estimate  of  the  vehicle 
position  is  used  to  predict  matchings  between  the  current  map  and  the  new  observed  features.  Second,  a  new  position  estimate  is 
computed  based  on  the  matchings  and  the  current  position  estimate.  Third,  the  map  is  updated  by  insetting  the  new  observations. 
This  involves  the  insertion  of  new  measurements  in  the  grid  representation,  the  insertion  of  the  new  features  in  the  map,  and  the 
updating  of  existing  map  features  that  have  been  matched  with  newly  extracted  features. 

We  have  iiKludcd  the  map  building  techniques  in  the  Camcgic-Mcllon  NAVIao  system.  A  terrain  map  was  maintained  over  a 
hundred  meters  while  the  vehicle  was  running  autonomously  under  control  of  the  road  following  program.  The  curreitt  vehicle 
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(a)  Terrain  map  Crom  one  image 
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(b)  Terrain  map  from  four  images 
Figure  5*1:  Terrain  map  building  (including  road  feanires) 

estimate  was  used  as  an  initial  estimate  for  the  matching.  Due  to  memory  limitations,  only  the  portion  of  the  map  within  a 
window  centered  at  the  cunem  vehicle  was  kept  in  the  system.  The  features  used  for  the  matching  pan  in  that  implementation 
are:  the  primitives  of  the  surface  description  (as  described  in  Section  3),  the  location  of  discrete  objects  (as  described  in  Section 
2\  and  location  of  the  road  edges  extracted  from  the  tcHcctance  channeL  The  features  are  weighted  according  to  their 
uncertainty,  for  example  the  variance  in  the  case  of  planar  features.  The  road  edges  are  a  special  type  of  features;  since  road 
detection  from  reflectance  is  currently  not  reliable,  the  edges  arc  used  only  if  the  contrast  in  the  current  reflectance  image,  e.g.  the 
strength  of  the  detected  road  edges,  is  high  enough. 


6.  Object  recognition 


6.1.  Recognition  strategy 

The  goal  of  an  object  recognition  algorithm  is  to  find  the  most  consistent  interpretation  of  a  scene  given  a  stored  list  of 
primitives  (A/;j  ,..,A#^),  the  model,  and  a  segmentation  of  the  observed  scene,  (•S’^j  ,..,Sy,).  The  algorithm  must  therefore 
search  all  the  possible  matchings  Tliis  search  being  an  combinatorial  problem,  the  main  issue  is  to 

prune  the  search  space  in  order  to  be  able  to  process  complex  scenes.  Many  strategies  have  been  proposed  for  solving  the  object 
recognition  problem  (1).  The  most  common  approach  is  to  use  geometric  constraints  to  constrain  the  search,  assuming  the  objects 
are  rigid.  This  approach  may  require  an  accurate  geometric  model  which  is  usually  not  available  in  our  application.  Another 
approach  is  to  generate  beforehand  the  possible  appearances  of  the  object  to  recognized  in  order  to  reduce  the  search  space  by 
precompiling  the  constraints  in  the  model  [8,  5].  Wc  use  a  combination  of  both  approaches  in  which  geometric  constraints  are 
precompiled  in  the  model.  The  model  has  two  components:  a  set  of  surface  patches,  M,  and  a  set  of  constraints,  C.  The 


constraints  encapsulate  knowledge  about  the  object's  shape,  such  as  "surfaces  and  Afy  are  orthogonal”.  A  constraint,  c, 
itsoci ated  with  a  set  of  regions  can  be  viewed  as  a  fiincdon  that  decides  whether  a  partial  matching 

,(Af^,5'y,))  is  acceptable.  The  number  n  of  regions  involved  may  be  different  depending  on  the  constraint.  For 
example  the  constraint  on  the  area  of  a  region  is  a  unary  constraint,  while  the  orthogonality  constraint  is  a  binary  constraint.  The 
list  of  constraints  and  their  implementation  arc  discussed  in  the  next  two  Seetions. 

The  search  algorithm  Hrst  constructs  a  list  of  candidates  for  each  model  region,  by  applying  the  unary  constraints 
associated  with  to  every  scene  region,  Sy  This  provides  a  first  reduction  of  the  search  space  according  to  unary  constraints. 
The  algositfam  then  explores  the  remaining  search  space,  discarding  the  pardal  solutions  that  do  not  satisfy  the  remaining 
ccnstraiius.  In  other  words,  each  time  a  new  pairing,  is  added  to  a  pardal  soludon,  .Jy,)....(Af^.5;,)).the 

constraints  associated  with  Af arc  evaluated  over  die  new  set  of  pairings.  The  partial  soludon  is  not  explor^  further  if  one  of  the 
constraints  is  not  satisfied.  The  result  of  all  the  constraint  evaluadons  are  stored  in  tables,  so  that  a  constraint  is  never  evaluated 
twice  on  the  same  set  of  pairings. 

The  result  of  the  search  algorithm  is  a  small  set  of  soludons.  The  last  step  of  the  recognidon  algorithm  is  to  compute  a  score 
that  reflects  the  quality  of  each  solution.  This  last  step  is  necessary  since  there  is  no  way  of  forcing  the  search  to  produce  only 
one  solution  because  of  near-symmctrics  in  the  model,  segmentauon  enors,  or  even  the  presence  of  several  instances  of  the 
object  in  the  scene.  The  actual  computadon  of  the  score  is  discussed  in  detail  in  Section  SJ.  The  next  sections  describe  the 
details  of  the  algorithm.  We  use  a  car  as  an  example  of  object  model  in  discussing  the  algorithm. 


6,2.  Constraints 

The  constraints  we  currently  use  are: 

•  NX,  NY,  NZ:  constrains  the  components  of  the  surface  normal  of  a  region.  This  constraint  is  used  to  implement 
natural  limitadons  on  the  orientation  of  an  object,  such  as  "the  roof  of  a  car  caiuiot  be  verdcal". 

•  Z:  constrains  the  verdcal  position  of  a  region. 

•  AREA:  constrain  the  area  of  one  region. 

•  ANGLE:  constrains  the  angle  between  two  regions. 

•  NEIGHBOR:  constrains  two  regions  to  be  neighbors  by  computing  the  distance  between  the  boundaries  of  two 
regions. 

•  EXCLUDE:  Forbids  two  regions  to  be  visible  at  the  same  dme.  This  constraim  is  based  on  the  nodon  of  aspects 
[8,  7].  An  aspect  is  a  set  of  regions  that  can  be  observed  from  a  given  viewpoint  Instead  of  explicitly  enumerating 
the  possible  aspects  of  an  object  the  EXCLUDE  constraint  describes  them  implicitly. 

•  DISTANCE:  constrains  the  distance  between  two  surfaces. 

The  constraints  ate  precomputed  and  stored  in  the  model.  Each  constraim  is  described  by  the  following  structure: 

•  number  of  arguments,  N:  For  example,  the  constraint /tlVGLE  which  constrains  the  angle  between  two  regions  has 
two  arguments.  The  maximum  number  of  arguments  is  currendy  three. 

•  evaluation  function,  F:  A  function  that  returns  an  interval  given  N  regions.  For  example,  the  constraint  ANGLE 
computes  an  interval  cemered  around  the  angle  between  two  input  regions.  The  size  of  the  interval  is  detemuned  at 
run-time  by  the  uncertainty  on  the  parameters  of  the  regions.  In  the  case  ANGLE,  the  interval  width  is  given  by  the 
angular  variance  within  the  two  input  regions. 

•  interval,  I:  An  interval,  or  set  of  intervals,  which  must  intersect  the  computed  interval  to  satisfy  the  constrainL 

This  representation  of  constraints  is  flexible:  A  new  type  of  constraint  can  be  easily  added  to  a  model  by  simply  defiiting  the 
appropriate  evaluation  function.  Building  a  new  model  is  easier  since  the  constraints  are  not  hardcoded  in  the  recognition 
program. 


6J.  Evaluatioa  of  the  solutions 

One  would  like  to  have  a  recognition  program  that  generates  only  one  solution  that  is  reported  as  the  recognized  object  in  the 
scene.  Unfortunately,  the  search  algorithm  generates  many  solutions  that  have  to  be  evaluated  in  order  to  determine  the  best  one. 
Ihere  are  three  reasons  why  the  search  generates  multiple  solutions:  First,  the  constraints  we  use  are  very  liberal  so  that  the  same 
model  can  be  used  for  a  wide  range  of  scenes,  consequently  false  solutions  are  difficult  to  avoid.  Second,  the  object  may  have 
near-symmctrics  that  lead  to  several  equally  valid  interpretations.  Third,  the  image  segmentation  being  imperfect,  an  object 
region  may  be  broken  into  several  pieces  in  the  image,  thus  producing  several  equivalent  solutioru. 

Our  approach  to  evaluating  solutions  is  to  first  compute  the  position  and  orientation,  or  pose,  of  the  object  for  each  solution,  to 
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ihoi  generate  a  synthesized,  or  predicted,  range  image  using  the  estimated  pose,  and  to  finally  coxielaie  the  predicted  image  tad 
the  original  range  image.  We  derive  a  score  Grom  the  correlation  measure  between  the  two  images  which  is  used  to  discard 
erroneous  solutions,  and  to  sort  the  other  solutions. 

The  pose  is  calculated  for  each  solution  by  minimizing  the  two  sums: 

-^j  II 

and 
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where  R  and  are  the  estimated  roution  and  translation,  nT  and  c  (resp.  and  are  the  surface  normal 

and  center  of  region  c  (resp.  y)  of  the  model  (resp.  scene).  The  summation  is  over  all  the  pairs  (scene  region  j,  model  region  0- 

hi  order  to  compute  the  predicted  image,  we  have  to  compute  the  position  of  each  point  of  the  model  in  image  coordinates,  as 
well  as  the  predicted  range.  The  position  in  the  predicted  range  image  of  a  point  ^  on  the  surface  of  the  model  is  given  by; 

.predicted  .  ,  ,  , 
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cof=(8^'*'-9,)/A0 
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In  this  ^uatioru  x,y,z  are  the  coordinates  of  the  transformed  point  Rp  -i- 1,  (row,  col)  is  the  predicted  location  is  the  image, 
ppttUcttd 

and  d  is  the  predicted  range  value  at  that  location. 

The  correlation  between  predicted  and  actual  range  images  is  given  by; 


.predicted  originai, 
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where  a.^  is  the  area  intercepted  by  pixel  and  the  summation  is  made  over  the  intersection  Fr\0  of  the  object  in  the 


predicted  image,  P,  and  the  object  in  the  observed  range  image,  O. 

The  correlation  must  be  normalized  to  obtain  a  score  5  that  is  between  0  and  1 ; 
S=<2rea(PnO)/area(PuO) 


X  (1  -  C/(K>tarea(Pr\0)))) 

In  this  equation,  IK  is  the  maximum  range  difference  allowed  between  predicted  and  observed  images  (independent  of  the 
image).  5  is  normalized  by  area(.Pr\0)/<irea(,PtjO)  to  avoid  problems  when  PnO  is  very  small,  in  which  case  we  would  give 
a  high  score  to  a  very  poor  solution.  Wc  use  the  score  5  to  eliminate  false  solution  (typically  5  <  0.5),  and  to  sort  the  solutions  by 
decreasing  score.  Figure  6-1  shows  the  solution  of  highest  score  found  on  one  image,  the  top  image  shows  the  superimposition 
of  the  recognized  model  and  the  range  image,  iltc  bottom  image  is  the  overhead  view  of  the  superimposition. 


6,4.  Results 

We  have  tested  the  object  recognition  program  on  23  images  of  two  different  objects,  each  image  corresponds  to  a  different 
aspect  of  the  objects,  or  to  a  different  distance  between  the  object  and  the  sensor.  Figure  6-2  shows  a  sample  of  the  set  of  images. 
The  failure  modes  of  the  program  arc  as  follows: 

•  In  two  cases,  the  program  failed  to  ftroducc  any  inunpretation.  These  cases  occur  when  not  enough  regions  have  been 
extracted  to  perform  the  recognition. 

•  In  two  cases,  the  program  produced  a  set  of  interpretations,  but  the  correct  solution  was  not  in  the  top-score  set  of 
solutions. 

•  In  three  cases,  the  program  produced  the  correct  interpretation  as  part  of  the  top-score  set  of  solutions  but  not  as  the 
best  solution.  These  cases  occur  when  a  near  symmetry  in  the  image  cannot  be  disambiguated  based  on  the  shape 
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(a)  Best  solution. 


(b)  Overhead  view. 
Figure  6*1:  Object  recognition 


information  only,  as  a  result,  both  the  correct  solution  and  its  symmetrical  have  the  same  score. 


Figure  6*2:  Object  recognition  on  a  sample  set  of  images 

The  object  recognition  algorithm  can  be  improved  in  several  ways:  The  recognition  of  one  object  requires  an  average  of  1,200 
lint  evaluations,  most  of  which  are  rarely  used  for  rejecting  a  partial  solution.  One  opdmizadon  is  to  order  the  constraints 
nodel  so  that  the  constraints  most  likely  to  prune  the  search  are  evaluated  first. 

scoring  procedure  leads  to  very  close  scores  for  symmetric  or  ambiguous  solutions.  As  an  example,  Hgure  6-3  shows  two 
itric  solutions  for  which  the  difference  between  the  two  scores  is  below  1%.  This  can  be  improved  by  including  other 
sources  of  information,  such  as  video  or  reflectance  images,  in  the  scoring  procedure. 
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7.  Conclusion 

We  heve  presented  a  sec  of  techniques  for  producing  3-D  environment  deseriptions  for  an  autonomous  vehicle.  Those 
techniques  and  their  output  descriptions  have  been  designed  to  fulfill  the  requirements  of  the  major  tasks  of  an  autonomous 
vehicle  such  as  obstacle  avoidance,  landmark  recognition,  and  map  building.  We  have  conducted  an  numerous  demonstrations  in 
the  CMU  autonomous  vehicle  system,  the  navlab,  for  navigation  and  map  building  applicadons. 

We  are  currently  pursuing  further  research  along  three  tines:  implementatiaa  on  faster  hardware,  handling  uncertainty,  and 
developing  a  more  general  scheme  for  object  recognition. 

The  primary  bottleneck  in  the  development  of  an  autonomous  vehicle  is  the  computation  time  required  by  the  sensory  modules 
such  as  the  range  image  analysis  module.  We  are  in  the  process  of  porting  the  algorithms  to  a  fast  systolic  machine,  the  warp 
[12].  The  obstacle  detection  module  has  been  already  successfully  demonstrated  with  a  cycle  on  the  order  of  one  secottd  on  the 
WARP.  The  second  line  of  work  is  the  representation  of  uncertainty.  Sensor  measurements,  vehicle  position  estimates,  and 
segmentation  process  induce  errors  in  the  final  3-D  description.  These  errors  can  be  quantified  and  taken  into  account  in  ail  the 
range  analysis  algorithms.  We  plan  to  use  an  expUcit  represenuuion  of  the  uncertainty  building  surface  description  (9, 2).  Third, 
we  plan  to  develop  a  more  general  scheme  for  object  recognition.  Third,  we  plan  to  apply  the  object  recognition  algorithm  to  a 
larger  class  of  objects  and  algorithms. 
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Abstract 

This  paper  describes  an  approach  to  robust  road  scene  interpretation  in  high-level  vision.  There  are  two  key 
ideas.  One  is  adjustable  explicit  scene  models  which  express  the  global  information  of  a  scene.  The  other  is  an 
interpretation  cycle:  evaluation,  modeling,  and  extrapolation,  which  is  the  mechanism  for  adjusting  the  explicit 
scene  models. 

First  the  interpreter  picks  up  some  tractable  segment  regions  and  generates  initial  hypotheses,  and  then  it  iterates 
the  interpretation  cycle  until  every  region  is  labeled  with  one  of  the  scene  objects  expected.  In  each  interpretation 
cycle,  labeled  regions  play  a  survival  game.  Those  that  are  consistent  get  their  plausibility  value  iitcreased.  Those 
that  are  not  get  their  plausibility  value  decreased  and  some  of  them  die  (become  unlabeled).  Surviving  regions 
propagate  their  labels  to  their  adjacent  similar  regions.  In  the  mean  time,  explicit  scene  models  are  also  adjusted 
with  the  latest  interpretation  in  each  cycle.  The  models  gradually  extract  global  information  out  of  the  scene. 
When  the  plausibility  values  of  the  models  become  high,  global  evaluation  starts  by  the  models  in  addition  to  local 
evaluation.  Some  deceptive  regions  are  easily  killed  by  the  global  evaluation. 

More  than  20  toad  scenes,  some  of  which  are  destroctively  shaded,  are  reliably  interpreted  by  the  implemented 
system  INSIGHT-IIL  From  the  view  point  of  high-level  vision,  inexhaustive  region  segmentation  in  low-level 
vision  is  also  suggested  and  discussed. 


1.  Introduction 


The  goal  of  this  study  is  to  build  a  robust  road  scene  interpreter.  Our  approach  is  to  intopret  components  such  as 
roads,  grass,  soil,  shadows,  sun  spots,  tree  trunks,  tree  foliage,  vertical  objects,  sky,  etc.,  in  a  road  scene  explicitly. 
We  focus  our  study  on  theoretical  aspects  of  natural  scene  understanding  rather  than  practicai  aspects  such  as  how 
quickly  a  road  is  extracted  in  order  to  navigate  an  actual  vehicle.  In  natural  scene  interpretation,  there  are  three 
substantial  problems:  low-level  region  segmentation,  intopretalion,  and  domain  knowledge  organization.  This 
paper  concentrates  on  the  second  problem,  interpretation. 

We  have  used  available  tools  for  low-level  region  segmentation.  The  segmentation  tool  is  histogram-based  and 
semi-automatic.  A  threshold  value  in  each  histogram  is  interactively  determined  by  an  operator.  Automatically 
segmented  images  are  poor  and  some  of  them  are  uninterpretable.  Insisting  on  fully-automatic  segmentation  could 
have  obscured  potential  problems  of  interpretation.  As  the  domain,  road  scenes  on  sidewalks  in  a  park  were  chosen, 
so  the  kinds  of  objects  in  a  scene  are  moderately  limited,  currently  10,  which  alleviates  the  problem  of  domain 
knowledge  organization.  On  the  other  hand,  complex  road  scenes  on  sidewalks  were  intendonaUy  selected  so  as 
to  uncovo-  the  potential  problems  in  interpretation.  Several  scenes  are  severely  shaded  by  trees.  A  couple  of  them 
are  not  usually  understood  even  by  a  human  without  predictions.  Another  intention  was  to  accumulate  knowledge 
out  of  many  scenes  (more  than  20  to  date).  This  was  also  requisite. 


Suppofud  by  a  gram  from  Sony  Coipontian. 
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Outdoor  natural  color  road  scenes  are  ill-stnictuied.  Trees  and  objects  often  shade  the  road  badly.  The  edges 
fCHined  by  shadows  and  sun  spots  are  much  stronger  than  those  of  roads.  Natural  objects  are  not  usually  made 
with  straight  lines.  Old  road  pavemoits  have  many  cracks  and  spots.  The  color  intensities  of  the  road  pavements 
vary  significantly  depending  on  their  surface  conditions,  the  weather,  the  direction  of  sun,  the  camera  condition, 
and  other  circumstances.  In  interpretation  of  natural  outdoor  color  scene,  there  are  two  main  questions: 

1.  How  to  extract  global  information  which  depicts  a  scene  roughly. 

2.  How  to  generate  reliable  initial  hypotheses. 

If  the  system  has  global  infcmnation  about  the  scene,  this  reduces  the  search  space  and  local  region  labeling 
becomes  unambiguous.  For  example,  if  the  system  knows  the  rough  location  of  the  road  in  a  scene,  only  the  regions 
along  the  rough  road  edges  are  ambiguous.  Mrae  reliable  road  edges  will  be  obtained  by  checking  intensity  steps 
crosswise  in  kmg  narrow  windows  along  the  expected  locations  of  the  edges.  Of  course,  global  information  is  not 
available  in  advance.  The  interpreter  itself  has  to  extract  it  In  our  approach,  explicit  scene  models  represent  global 
information  and  the  interpretation  cycle  is  a  mechanism  ftv  adjusting  them. 

Good  initial  hypotheses  usually  lead  to  successful  interpretation  while  bad  initial  raies  often  do  not  It  is  hard 
to  change  the  labels  of  the  initially  labeled  regions.  This  is  natural  because  initially  labeled  regions  are  the  seeds 
fcv  the  following  interpretation.  This  ptoUem  is  partly  beyond  this  paper,  because  it  also  depends  on  the  low-level 
region  segmentation.  However,  several  ideas  for  genertaing  reliable  initial  hypotheses  ate  described  in  Section  62.1. 
Inexhaustive  region  segmentation  in  low-level  vision  is  also  suggested  and  discussed  in  Section  7.1. 

The  major  contribution  of  this  study  by  exftiicit  scene  models  and  an  interpretation  cycle  is  enumerated  as 
follows: 

•  Global  information  of  a  natural  scene  can  be  extracted  with  them  and  make  interpretation  reliable. 

•  They  eliminate  nrost  of  the  search  space  and  interpret  oversegmented  and  fiagmented  regions  consistently. 

•  To  some  extent,  they  can  cope  with  the  proUem  of  undersegmented  regions. 

•  Cyclic  interinetation  mechanism  is  appropriate  for  intensive  interaction  between  high-level  and  intermediate- 
level. 

The  following  Section  2.  explains  pioneering  worit  in  natural  scene  interpretation.  Section  3.  illustrates  the  main 
idea  of  this  study:  e;q)licit  scene  models  and  an  interinetation  cycle.  In  S^tion  4.,  the  ovmview  of  the  developed 
system,  INSIGHT-m  is  presented.  In  Section  5.,  the  results  of  1 1  images  ate  shown.  Three  examples  a(  the  explicit 
scene  model  behaviors  follow.  Section  6.  goes  into  the  detail  of  INSIGHT-m.  Possible  enhancements  are  explained 
in  Section  7.  Section  8.  concludes  this  study. 


2.  Related  Work 


Dnpet  [1][2]  has  experimented  with  knowledge-based  road  scene  interpretation  in  the  "Schema  System”  architec¬ 
ture,  which  originates  from  the  VISIONS  system  P].  His  system  comprises  indq)endent  object-specific  interpreters, 
called  schemata,  and  a  central  black  board  for  communication  among  them.  The  system  interprets  a  road  scene  m 
a  u^Hlown  manner  by  starting  from  the  toad-scene  schema  and  then  calling  sulqiart  schonata  such  as  roof,  sky, 
road,  and  tree  hierarchically.  Each  schema  generates  initial  hypotheses  based  on  cokv,  texture,  shape,  area,  and 
position  of  segmented  regitms.  Then  it  conoborates  or  denies  them  by  checking  positive  and  negative  evidence, 
extending  hypotheses,  resolving  conflicts,  and  calling  subpart  schemata.  Information  about  the  latest  interpretation 
with  confidence  values  is  exchanged  through  the  central  blackboard. 


47 


Ohta  [4]  has  built  a  knowledge-based  outdoor  color  scene  inteqneter  whkh  recognizes  four  objects  and  two 
subobjects:  sky,  trees,  buildings  with  subobjects  (windows),  and  roads  with  subobjects  (cars)  in  a  combined  bottom- 
up  and  top-down  manner.  In  his  system,  each  large  region  initially  has  four  hypotheses:  sky,  a  tree,  a  building,  and 
a  road  with  OHiecmess  values  which  are  modified  by  evaluatioa.  The  hypothesis  which  gets  the  highest  correctness 
value  wins.  It  first  tentatively  merges  small  regions  with  one  of  their  large  neighbOT  regions  and  then  calculates 
the  four  conectness  values  (k  the  hypotheses  on  each  large  region  by  checking  coIot,  texture,  slvqre,  position  of 
segmented  regions,  relations,  contrast,  lines,  etc.,  in  a  bottom-up  manner.  After  this,  these  hypotheses  are  evaluated. 
Then,  the  interpretation  goes  to  rule-based  top-down  analysis,  which  tries  to  select  the  most  appropriate  label  for 
each  large  region  and  find  labels  for  smaller  regions.  Whenever  the  top-down  analyzer  makes  a  significant  decision 
such  as  the  position  of  the  scene  horizon,  the  interpretafiCMi  goes  ba<±  to  the  boomn-up  process  and  re-evaluates 
the  labels  for  large  regions.  In  this  way.  the  bottom-up  process  and  u^r-down  process  work  cooperatively. 

McKeown  [S][6]  has  demonstrated  a  rule-based  system.  SPAM,  with  approximately  SOO  OPSS  rules  for  aoial 
airpot  imagery  containing  terminals,  parking  aprons,  hangars,  runways,  taxiways,  parking  lots,  roads,  tarmac, 
grassy  areas,  and  fimctional  areas.  SPAM  is  a  region-based  bottom-up  system  with  top-down  predictions.  It  allows 
multiple  initial  hypotheses  on  each  region,  which  are  derived  by  examining  local  propmies  such  as  2-D  shape, 
location,  (mentation,  texture,  area,  and  perimeter.  Then  it  checks  consistency  locally,  groups  local  interpretations 
into  functional  areas,  and  generates  one  (v  a  few  airpcm  mcxlels.  Height  information  extracted  by  stereo  images 
is  also  utilized  when  necessary.  Recently  he  also  devekqred  knowledge  acquisition,  machine  rule  generation,  and 
performance  analysis  tools  [7]  in  order  to  ease  and  expe^te  knowledge  acquisition  processes  [8].  Knowledge  for 
suburban  house  scenes  was  collected  in  the  SPAM  architecture  with  these  tools. 

Brooks  [9]  has  developed  a  viewpoint-insensitive  3-D  model-based  system  called  ACRONYM,  which  was 
applied  to  jet  plane  recognition  in  aerial  photographs.  ACRONYM  has  an  intermediate  level  rqnesentation  with 
ribbons  (trapezoids  with  long  height)  and  ellipses,  to  which  both  3-D  nuxlel  and  low-level  edges  are  transfimned 
The  interpretation  takes  place  in  this  level  in  a  top-down  maruier.  Recently,  ACRONYM  was  transferred  to  industry 
[10]  and  uncertainty  management  [11]  has  been  incorporated  into  it 

A  survey  by  Binford[12]  discusses  model-based  irttage  analysis  systems  including  work  by  Nagao  and  Mat¬ 
suyama  [13],  Ohta  [4],  Brooks  [9],  and  Shirai  [14]. 


3.  Adjustable  Explicit  Scene  M(xlels  and  Interpretation  Cycle 


Adjustable  explicit  scene  models  and  interpretation  cycle  are  the  two  key  ideas  of  INSIGHT-m  for  extracting  global 
information  out  of  a  interpreted  scene.  The  global  i^cmnation  makes  global  evaluation  possible  and  tiius  leads  to 
reliable  consistent  interpretaticxi.  In  scene  interinetation,  a  chicken  and  egg  problem  has  arisen:  a  scene  is  hardly 
interpreted  locally  and  ctnrectly  without  global  infcmnation  of  the  scene;  the  global  information,  on  the  other  hand, 
can  not  be  obtained  withrxit  lo^  interpretation  fragments.  These  ideas  are  an  q;)proach  to  this  problem. 

At  first,  the  interpreter  does  not  have  global  information  of  a  road  scene  at  all,although  usuaUy  there  are  some 
tractable  regions  which  are  often  large,  occupy  a  lower  portion  of  the  image,  have  typical  col(ns,  and  can  be  labeled 
in  a  bottom-up  manner.  Once  even  just  a  few  regions  are  labeled,  they  tell  a  little  about  the  scene.  Very  rough 
global  information  of  the  scene  can  be  extractecL  Of  course,  this  global  information  is  itot  reliable,  but  better  than 
nothing.  It  can  be  used  skeptically  in  the  following  interpretation  stage.  Gradually,  mote  regions  are  labeled  and 
thus  the  extracted  global  information  is  improved.  If  this  global  information  feedback  is  reascmable,  interpretation 
converges  to  an  appropriate  one. 


3.1.  Adjustable  Explicit  Scene  Models 

Adjustable  explicit  scene  models  explicitly  approximate  the  tqrpearances  of  the  objects  in  a  road  image  with  simple 
geometrical  figures.  Those  mcxiels  have  four  or  five  degrees  of  fieedom  and  ate  adjusted  to  the  latest  interpretation 
in  each  interpretation  cycle. 
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Rgine  1  (a)  dqricts  a  triangular  e>q>licit  scene  model  for  the  appearance  of  a  road  or  a  long  narrow  stripe  on  an 
assumed  flat  ground  {flane.  It  has  four  degrees  of  freedom:  the  vanishing  point  moves  horizontally  and  vertically; 
lower  ends  the  right  and  left  edges  move  horizontally.  *  Figure  1  (b)  is  a  parallelogtam  explicit  scene  model  ^ 
for  a  nee  trunk  or  a  vertical  objea  protruding  Grom  the  ground.  Since  its  upper  and  lower  bases  remain  horizontal, 
it  has  5  d^rees  of  freedom:  location  (2  degrees  of  freedom),  height,  width,  and  lean. 


(a)  Road  Model 


(b)  Tree  Trunk  Model 

Rgute  1:  Adjustable  explicit  scene  models 

Figure  2  illustrates  how  the  road  model  is  adjusted  to  an  inteqneted  image,  lb  measure  the  igjproptiatBness 
of  the  road  model,  a  discrepancy  value  is  defined:  the  discrqjancy  for  the  road  model  is  the  sum  of  the  road  area 
outside  the  model  +  the  giass/s(ti  area  inside  the  modeL  The  toad  model  is  adjusted  such  that  the  discrepancy 
value  is  locally  minimized.  Note  that  the  erq>licit  scene  models  ate  not  interfered  by  jagged  region  boundaries. 

Figure  3  shows  the  tree  trunk  model  adjustment  The  discrepancy  value  the  tree  trunk  model  is  the  sum 
of  the  tree  trunk  area  outside  the  model  +  die  non-tree-trunk  area  inside  the  model  The  tree  trunk  model  is  also 
adjusted  to  the  local  minimum  cS  the  disctqnncy  value.  Because  of  this,  it  is  not  affected  by  the  trunk  areas  of 
other  trees.  Hnding  the  local  minimum  of  the  discrepaiK^  value  is  not  very  expensive.  It  only  requires  checking 
the  pixels  along  the  model  boundaries.  Section  6.5.  explains  the  details. 

In  the  eariy  stages  of  interpretation,  some  regions  are  labeled  and  others  are  not  But  the  explicit  scene  models 
are  adjusted  by  using  partially  labeled  regions.  Though  the  adjusted  models  may  be  less  reliable,  they  can  grasp 
very  tough  information  of  the  scene.  For  example,  unlabeled  regions  in  the  middle  of  the  adjusted  rood  model  are 
likely  to  be  fragments  of  the  toad,  and  those  away  from  the  adjusted  rood  model  are  likely  to  be  fragments  of  the 
grass  or  the  soil.  If  the  plausibility  value  of  the  model  becomes  higher  for  some  teastm,  the  model  can  classify  the 


‘The  roed  model  cn  be  expressed  with  soother  oombinstiau  of  peiietets.  For  exen|de,  vMiishiiis  poim,  road  diiectioa,  and  road  width, 
which  also  has  4  degrees  of  freedom. 

^TlMtiih  these  scene  models  are  2  dimensianel  they  may  be  derired  from  3  dimensional  scene  modds  by  perspective  projectian. 


The  model  has  4  degrees  of  freedom 
Discrepancy  >  road  area  outside  the  model 
grass/soil  area  inside  the  model 


(a)  Before  Adjustment 


f  b)  Adjusted  Road  Model 


Figure  2;  Road  model  adjustment 

unlabeled  regions  and  even  discard  some  contradicting  labels.^ 

These  explicit  scene  models  are  simple  and  cannot  mcpress  the  exact  shapes  or  the  details  of  roads  and  tree 
trunks,  such  as  curves  and  tree  branches.  Complex  or  flexible  models  with  more  degrees  of  freedom  may  be  applied 
aftowards  if  necessary.  With  the  global  information  extracted  by  the  simple  models,  those  complex  or  flexible 
models  will  be  adjusted  quickly  and  reliably  to  the  scene. 

An  interesting  aspect  of  the  explicit  scene  models  is  that  they  help  detect  misinterpretation.  The  way  wide  or 
narrow  road  is  strange.  The  very  high  tn  low  vanishing  point  is  uncommon.  If  a  tree  trunk  is  found  in  the  middle 
of  the  road,  it  implies  something  strange  has  happened.  The  system  can  easily  determine  unusual  situations  by 
checking  the  adjusted  models.  Then  it  will  be  able  to  retry  the  interpretation  of  the  scene  with  different  parameters 
in  low  level,  or  activate  a  contingency  planner  which  may  decide  to  decelerate  the  autonomous  land  vehicle  and 
take  an  image  with  another  camera  iris  or  angle.  The  acceptable  range  of  the  adjusted  models  will  be  detomined 
heuristically  or  by  the  interpretation  result  of  the  previous  scene. 


3.2.  Interpretation  Cycle 

Figure  4  shows  the  interpretation  cycle  of  the  high-level  scene  interpreter.  It  consists  of  an  initial  hypothesis 
generator,  an  evaluator,  a  modeler,  and  an  extrsgxtlattH'.  In  general,  the  interpretation  proceeds  as  follows.  First, 
the  initial  hypothesis  generator  picks  up  some  tractable  regions  by  checking  their  properties  (areas,  locations, 
intensities,  adjacent  regions’  intensities,  shapes,  straight  lines,  parallel  lines,  trapezoidal  lines,  etc.)  and  makes 
initial  hypotheses.*  Then,  the  inteipretation  cycle  starts  and  interates  until  every  region  is  labeled  with  one  of 
the  objects  expected.  The  evaluator  checks  local  consistencies  of  the  current  intopretation  by  basically  looking 
at  pairs  of  labels  of  adjacent  regions.  If  they  are  ctmsistent,  their  plausibility  values  are  increased,  if  they  are 
not,  their  [flausibility  values  are  decreased.  Every  region  with  negative  plausibility  value  is  unlabeled.  When 
the  plausibility  values  of  the  models  become  higtiCT,  evaluator  starts  checking  global  consistencies  of  the  current 
interpretation  by  looking  at  labels  and  adjusted  models.  If  a  label  contradicts  a  model,  its  plausibility  value  is 

^Thif  actiaii  might,  in  tom,  require  model  adjunmeau. 

^biitial  hypodietit  geneiator  it  more  oigeniaed,  which  it  described  m  Section  6.2. 


50 


The  model  has  S  degrees  of  freedom 
Discrepancy  -  trunk  area  outside  the  model 
*  non-trunk  area  inside  the  model 


(a)  Before  A(^ustment 


(b)  Adjusted  Trunk  Model 


Figure  3:  Tree  trunk  model  adjustment 


decreased  drastically.  Then,  the  modeler  adjusts  the  ex|^it  scene  models  to  the  current  interpretation,  one  model 
after  another.  It  generates  necessary  models  and  occasionally  eliminates  inappropriate  models.  The  extrapolate 
picks  up  some  moderately  tractable  unlabeled  regions,  that  are  adjacent  to  labeled  regions  and  makes  new  hypotheses 
by  examining  their  (noperties  and  labeled  neighbor  regions.  Finally  the  control  goes  back  to  the  evaluatiei  and 
starts  the  next  interpretation  cycle. 

Figure  S  illustrates  the  interpretation  cycles  by  a  simplified  example,  which  has  only  a  region  map,  color  data, 
and  four  interpretation  cycles.  Figure  S  (a)  is  the  region  map  with  10  segmented  regions.  Figure  S  Q>)  shows  the 
colors  of  the  regions.  It  iiKludes  one  ambiguous  region  and  two  deceptive  regions.  R9  is  ambiguous  since  its  color 
is  in-between.  R6  and  RIO  are  deceptive  because  their  colms  differ  from  those  of  ttie  neighbor  regions.  The  initial 
hypothesis  generator  picks  up  the  tractable  regions  Rl,  R4,  RS,  and  R7  and  makes  hypotheses:  sky,  grass,  road, 
and  grass,  respectively.  Then,  the  interpretation  cycle  starts. 

In  the  first  cycle,  the  evaluator  slightly  raises  the  plausibility  values  of  Rl,  R4,  and  RS  by  checking  the 
consistency  between  adjacent  labeled  regions.  The  modeler  then  adjusts  the  triangular  road  model.  The  adjusted 
toad  model  is  far  from  ideal  since  its  vanishing  point  is  too  high,  but  it  has  obtained  very  tough  global  information 
of  the  road.  Then,  the  extrapolator  is  activated-RS  {Hopagates  to  R8,  R7  labels  R2  with  grass.  Both  R4  and  R5 
try  to  propagate  to  R9  since  its  color  is  between  thOTi.  R6  becomes  road  and  RIO  becomes  grass,  oroneously. 
In  the  second  cycle,  many  of  the  regions  have  been  labeled,  but  R6,  R9,  and  RIO  are  mislabeled.  The  evaluator 
raises  the  plausibility  values  of  the  labeled  regions,  accept  those  mislabeled.  Because  the  plausibili^  value  of  R9 
becomes  negative,  R9  is  unlabeled.  Then  the  modeler  adjusts  the  model  and  gets  much  better  global  information. 
The  vanishing  point  is  almost  cmrecL  The  plausibility  level  of  the  model  is  now  usable.  Then,  the  extrapolator 
partially  uses  the  model  and  RS  propagates  to  R3  arai  R9.  In  the  third  cycle,  all  regions  except  deceptive  ones 
are  correct.  R9  survives  the  evaluation,  but  the  deceptive  regions  are  rejected  by  using  adjusted  road  models.  The 
modeler  extracts  the  proper  road  location  and  the  plausibility  level  of  the  model  becomes  reliable.  In  the  following 
octrapolation,  the  deceptive  regions  are  appropri^y  labeled  with  the  reliaUe  road  model.  In  the  fourth  cycle, 
every  region  survives  the  evaluation.  The  road  model  does  not  change  in  the  modeling  step  attd  the  interpretation 
ends. 

Unlike  some  other  region  based  interpretation  systems  (4][S],  INSIGHT-m  gives  a  segmented  region  at  most 
one  label  of  the  scene  objects  at  a  time.  With  this  restriction,  the  otplicit  scene  models  can  be  adjusted  uniquely. 
Since  this  restriction  temporarily  excludes  alteniative  labels,  the  evaluator  has  a  local  and  global  negative  evaluation 
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Figure  4:  Interpretation  Cycle 


mechanism  which  refuses  inconsistent  labels  and  gives  chances  to  the  alternative  labels. 

The  robustness  of  the  interpretor  resides  in  a  balance  among  four  modules  depicted  in  Figure  4.  None  of  the 
modules  is  perfect  The  four  modules  woilc  together  to  compensate  each  other.  Some  strong  rules  in  a  module  may 
work  effectively  on  a  particular  road  scene,  but  not  on  other  scenes.  Rules  should  be  simple  and  straight-forward 
so  that  the  obvious  regions  are  labeled  quickly.  Occasional  mistakes  are  taken  care  of  by  the  other  modules.^  For 
example,  the  initial  hypothesis  generator  should  label  a  few  of  the  easiest  regions  but  can  leave  sli^tly  ambiguous 
regions  unlabeled,  since  the  following  interpretation  cycle  is  ctg)able  of  disambiguating  diem.  The  exuapolator  can 
make  easy  hypotheses  since  the  negative  evaluatkm  is  ^le  to  reject  bad  hypotheses.  The  evaluator  do^  not  have 
to  be  nervous  because  the  modeler  is  obtaining  global  information  and  will  settle  the  interpretation  at  the  end.  In 
the  case  of  a  very  difficult  scene,  the  interpretation  does  not  converge.  But  it  can  be  detected  by  monitoring  the 
number  of  interpretation  cycles  itoated  and  the  plausibility  level  of  the  models.  Then  a  contingency  planno'  may 
be  activated. 


4.  System  Overview 


The  knowledge-based  road  scene  interpretation  system,  INSIGHT-in,  is  capable  of  extracting  roads,  grass,  soil, 
shadows,  sun  spots,  tree  trunks,  tree  foliage,  vertical  objects,  sky,  and  unknown  without  predictions  excqit  for  the 
existence  of  the  road  and  the  rough  cocrespondeiKe  between  a  horizontal  line  in  an  image  and  an  actual  scene 
iKuizon.  It  also  detects  pairs  such  as  a  tree  trunk  and  its  shadow,  the  direction  of  sun  (right  or  left),  and  the 
weather  (sunny  or  not),  and  uses  this  infarmatkm  in  interpretation.  By  inverse  projection  (back  projection),  it 
goierates  a  rc^  map  of  the  scene  although  its  depth  information  is  relative.  Its  high-level  interpretor  has  187 
OPS83  [IS]  (descendant  of  OPS5  [16])  rules  and  C++  [17]  functions  for  model  adjustment  The  number  of  rules 


^Actually,  the  implenMated  fynem  freqnendy  dungei  iu  mind.  Once  it  wu  observed  that  a  regian  was  rdabekd  four  times  and  finally  the 
fowth  hypothesis  stuvived. 
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(a)  Region  Map  (b)  Color 


W:White 

Q:Green 

B:Blue 

BG:Blue-Green 


(c)  Initial  Hypotheses 


Figure  5;  A  sim^dified  example  of  interpretation 
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fired  is  iqqnoximately  2000  to  5000  for  one  road  scene.^  The  system  runs  on  SUN-3/260  workstations.  One 
road  scene  takes  about  30  to  60  minutes  to  interpret,  depending  on  the  complexity  d  the  scene.  An  experimental 
fuUy-automatic  low-resolution  version  of  the  system  interprets  it  in  around  eight  minutes.  10  road  scenes,  some 
badly  shaded,  are  interpreted  by  the  system.  In  addition  to  these,  more  than  IS  road  scenes  are  intoineted  by  the 
fuUy-automatic  low-resolution  version  of  the  system.  Their  high-level  knowledge-based  interpreters  are  exactly  the 
same. 

Hgure  6  shows  the  system  organizadoa  The  color  median  filter  described  in  Appendix  A,  which  considers 
color  pixels  as  vecttvs  and  keeps  color  edges  sharper  than  the  scalar  median  filter,  is  first  applied  to  a  raw  color 
road  scene  image  of  512  x  480  in  spatial  resoludtMi  with  8  bit  pixel  depth.  Second,  the  noise  reduced  image 
is  semi-automatically  and  recursively  segmented  by  a  histogram-based  region  segmentation  tool,  PHOENIX  [18] 
[19].  At  each  step  PHOENIX  calculates  red,  green,  and  blue  histograms  of  a  scene  portion,  and  a  human  selects 
one  histogram  and  detomines  a  threshold  value  by  lotdting  for  the  deepest  valley  in  it  Then  the  scene  region  is 
divided  into  several  smaller  regions  by  using  the  t^slu^d  value.  This  process  recurs  until  all  the  regions  become 
homogeneous  or  small.  Third,  the  color  edge  detector  [20]  extended  from  the  Canny  edge  detects  [21]  extracts  edge 
information  from  the  raw  image.  Fourth,  the  feature  extraction  program  calculates  the  intermediate  representation 
of  each  region  out  of  the  region  map,  the  edge  data,  and  the  raw  image.  The  intermediate  representation  includes 
the  region  map,  mean  intensities,  standard  deviations,  texture,  area,  perimeter,  mass  center,  minimum  bounding 
rectangle  (MBR),  scatter  matrix,  boundaries,  vertices,  lines,  and  relations  among  regions,  boundaries,  vertices,  and 
lines.  Fifth,  the  extracted  intermediate  representation  data  are  fed  to  the  high-level  Imowledge-based  interprets  and 
the  road  scene  is  interpreted.  Last,  the  adjusted  scene  models  proceed  to  the  map  generation  and  a  road  map  is 
drawn  by  inverse  projecticML 


Figure  6;  System  (xganization 

In  the  case  of  the  fuUy-automatic,  low-resolution  venkm  trf  the  system,  only  the  region  segmentation  tool  is 
different  Instead  of  PHOENIX,  a  region  segmentation  tool  [22]  based  on  the  distances  among  pixels  in  cdor  space 
are  utilized.  It  generates  a  region  map  (rf  64  x  60  in  spatial  resolution  by  grouping  similar  pixels.  This  version  wiU 
be  integrated  with  an  actual  AutotKMnous  Land  Vehicle  [23]  [24]  developed  at  Carnegie  MeUon  University. 

Figure  7  Ulustrates  the  high-levd  knowledge-based  interpreter  configuration.  The  intermediate  reinesentation 
data  enter  the  working  memory  d  the  rule-based  system.  The  rules  d  the  initial  hypothesis  generator  are  activated 

*Becante  of  the  cyclic  mechaiiiim,  the  aoihber  at  rale  6riii(>  if  mncfa  leifer  than  the  number  of  ralei .  This  it  an  unccmman  chancteriitk 
of  the  tyiteni  at  oin»oeed  to  oidinafy  expen  lyiienit. 
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and  label  some  tractable  regions.  Then,  the  rules  of  the  evaluator,  the  modeler,  and  the  extrapolator  examine 
the  scene  intrapietation  data  cyclically  until  every  region  is  labeled.  All  the  intermediate  representation  data,  the 
temporal  data,  the  explicit  scene  models,  and  the  latest  interpretation  expressed  as  an  iconic  512  x  480  array  of 
labels  reside  in  the  woildng  memory. 


Figure  7;  High-level  knowledge-based  interpreter  configuration 


5.  Results  of  the  Road  Scene  Interpretation 


Figures  8~18  show  the  results  of  road  scene  interpretation  by  INSICHT-m.  Figures  8~11  are  the  shaded  roads. 
Figures  12~15  have  trees  in  the  scenes.  In  Figure  16,  there  are  two  vertical  objects  (two  men).  Figure  17  is  an 
intersection  with  trees.  Figures  8~18  (a)  are  original  road  scenes.  Figures  8~18  (b)  show  their  region  segmentation. 
Regions  are  painted  with  their  identification  numbers.  Hgures  8~18  (c)  tell  their  solution  class.  Regions  shaded 
with  vertical  lines  mean  above  ground  solution  class.  Those  shaded  with  horizontal  lines  are  on  the  level  of  ground 
solution  class.  Figures  8'''18  (d)  illustrate  their  interpretation.  Each  of  the  10  objects  is  painted  with  the  same 
intensity  and  texture.  Road  regions  are  labeled  (noperly,  and  thus  the  road  looks  flat  except  for  the  very  dark 
shadows  and  sun  spots.  In  Figures  8~18  (e),  adjusted  scene  models  are  drawn  on  segmented  regions  painted  with 
their  mean  intensities.  They  are  extracting  the  location  of  the  road  appmpiately.  Goieraied  im^  are  depicted  in 
Figures  8^18  (f).  Roads  are  expressed  with  pairs  of  parallel  lines,  and  trees  or  vertical  objects  are  drawn  with 
circles. 

Because  of  different  camera  calibration,  the  images  of  Figure  12and  Figure  15  look  blue  and  those  of  Figures  8~ 
11  and  Figures  14»-15  look  red.  Figure  12  and  Figure  13  were  digitized  from  VCR  tapes.  Their  original  images 
are  blurred  and  very  noisy.  Some  of  the  vanishing  points  are  in  the  scene  while  others  are  beyond  the  top  of  the 
image  frame.  Segmented  regions  are  not  always  extracting  proper  shapes.  Their  boundaries  are  jagged.  Thoe  are 
many  decqjtive  and  ambiguous  regions  whose  properties  are  very  different  from  typical  ones. 

Figure  10  and  Fgure  11  are  very  difficult  scenes  because  of  the  destructive  shadows.  Even  a  human  cannot 
understand  the  scene  in  Figure  11  without  predictions.  To  interpret  those  road  scenes,  INSIGHT-m  needs  more 
interpretation  cycles.  The  road  models  are  not  stable  at  first  and  take  a  longer  time  to  converge. 

Fgures  19~21  trace  the  behavior  of  the  modeb  which  have  been  shown  in  Figure  9,  Figure  14,  and  Figure  17, 
respectively.  The  initial  hypothesis  generator  is  skeptical  and  makes  only  several  hypotheses  as  are  shown  in 
Figures  19~21  (a). 
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(c)  Solution  class 


(d)  Interpretation 


(e)  Adjusted  model  (0  Generated  map 


Figure  8:  Shaded  road  1  (53  regions,  103  boundaries,  97  vertices,  and  332  lines) 


(e)  Adjusted  model  (0  Generated  map 


Figure  9:  Shaded  road  2  (122  regions,  299  boundaries,  229  vertices,  and  139  lines) 


(c)  Solution  class 


(d)  Interpretation 


(b)  Region  segmentation 


(e)  Adjusted  model 


(0  Generated  map 


Figure  10:  Shaded  road  3  (113  regions,  271  boundaries,  226  vertices,  and  169  lines) 
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(a)  Original  image 


(b)  Region  segmentation 


(e)  Adjusted  model  (f)  Generated  map 


Figure  11:  Shaded  road  4  (121  regions,  3QO  boundaries,  233  vertices,  and  291  lines) 
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(b)  Region  segmentation 
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Figure  12:  Road  with  trees  1  (21  regions,  30  boundaries,  33  vertices,  and  326  lines) 


(a)  Original  image 


(b)  Regirni  segmentation 


(c)  Solution  class 


(d)  Interpretation 


(f)  Generated  map 


Figure  13;  Road  with  trees  2  (95  regions,  255  boundaries,  213  vertices,  and  543  lines) 
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(b)  Region  segmentation 


(a)  Original  image 


(c)  Solution  class 


(d)  Interpretation 


(e)  Adjusted  models 


Figure  14:  Road  with  a  tree  3  (78  regions,  194  boundaries,  155  vertices,  and  133  lines) 
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(e)  Adjusted  models 


(b)  Regicm  segmentation 


(a)  Original  image 


(d)  Intnpietation 


(c)  Solution  class 


nerated  map 


Figure  15:  Road  with  a  tree  4  (96  regions,  234  boundaries,  187  vertices,  and  96  Imes) 


(b)  Region  segmentation 


(a)  Original  image 


(d)  Interpretation 


(c)  Solution  class 


(f)  Generated  map 


(e)  Adjusted  models 


Figure  16:  Road  with  vertical  objects  (54  regions,  116  boundaries,  98  vertices,  and  228  lines) 
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(c)  Solution  class 
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(d)  Interpretation 
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(e)  Adjusted  models  (f)  Generated  map 

Figure  17:  Intersection  (39  regions,  67  boundaries,  67  vertices,  and  159  lines) 
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(a)  Original  image 


(b)  Region  segmentation 


(c)  Solution  class 


(d)  Interpretation 


(e)  Adjusted  model 


(f)  Generated  map 


Figure  18:  A  full-automatic  system  example  (18  regions,  32  boundaries,  43  vertices,  and  105  lines) 
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(g)  Fourth  interpretation  cycle 


"fiiplpiiiliii 


'''^5W::;-Ai;j':':^<?>.:'<:^ici::::::t':! 


(i)  Fifth  interpretation  cycle 
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(h)  Adjusted  road  model  4 
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(j)  Adjusted  road  model  5 
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(k)  Sixth  interpretation  cycle 

Figure  19;  Behavior  of  a  road  model 
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(1)  Adjusted  road  model  6 


(a)  Initial  Hypotheses 


(b)  Adjusted  models  1 


lecond  interpretation  cycle 


(d)  Adjusted  models  2 


(e)  Third  interpretation  cycle 


(0  Adjusted  models  3 


Figure  20:  Behavior  of  scene  models 
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(g)  Fourth  interpretation  cycle 


(1)  Adjusted  models  6 


(i)  Fifth  interpretation  cycle 


(k)  bixth  interpretation  cycle 


(j)  Adjusted  models  5 


(h)  Adjusted  models  4 


Figure  20:  Behavior  of  scene  models 
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(b)  Adjusted  intersection  model  1 


(a)  Initial  Hypotheses 


(d)  Adjusted  intersection  model  2 


(c)  Second  intensretation  cycle 


(f)  Adjust^  intersection  model  3 


(e)  Third  interpretation  cycle 


Figure  21;  Behavior  of  a  intersection  model 


(h)  Adjusted  intersection  model  4 


(g)  Fourth  interpretation  cycle 


0)  Adjusted  inteisection  model  S 


(i)  Fifth  interpretation  cycle 


(k)  Sixth  interpretation  cycle 


(1)  Adjusted  intersecQon  model  6 


Figure  21:  Behavior  of  a  intersection  model 
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6.  The  Road  Scene  Interpretation  System  in  Detail 
6.1.  Feature  Extraction  and  Intermediate  Representation 

The  quality  of  the  semi-automatic  legiaa  segmentation  by  PHOENIX  is  better  than  fiilly-automatic  segmentation, 
but  sM  far  from  ideal.  Especially  when  the  image  is  sha^  and  complicated,  it  (rften  generates  hair-like  long  and 
narrow  portions  extending  from  the  bodies  along  edges.  Some  regions  are  not  separated  and  some  are  firagmented. 
The  boundaries  are  jagged.  In  the  feature  extractimi,  first  the  region  trinunm'  is  executed  on  the  region  map  to 
alleviate  these  problems.  It  breaks  hair-like  portions,  cuts  bridges  ctmnecting  two  big  bodies,  and  then  meiges 
each  small  noisy  fragment  to  the  adjacent  region  which  shares  the  longest  boundary.  After  region  trimming,  the 
intermediate  representation  data  are  calculated.  The  data  structures  of  the  intermediate  representation  are  similar  to 
those  of  the  work  by  Ohta  [4]. 

In  addition  to  the  region  rrug),  regions,  boundaries,  vertices,  and  lines  are  the  primary  elements  of  the  intermediate 
representation.  Their  structures  are  shown  in  Figure  22,  Figure  23,  Figure  24,  and  Figure  25.  The  relation  among 
regions,  boundaries,  vertices,  and  lines  are  depicted  in  Hgure  26.  A  region  is  enclosed  by  sevraal  boundaries;  each 
one  of  them  is  terminated  by  two  vertices.  A  line  is  littked  to  the  nearest  boundary. 

struct  region{ 

region  identificatian  number 

number  of  neighbors 
neighbOT  identification  numbers 
number  of  holes 
hole  identificatkm  numbers 
number  of  boundaries 
boundary  identification  numbers 

mean  intensity 

(Black&White,  R,  G,  B,  R-B) 
standard  deviation 

(Black&White.  R,  G.  B,  R-B) 
texture  value 

area 

perimeter 

mass  center  tow,  coluitm 
minimum  bounding  rectangle  (MBR) 
towmin,  towmax,  colmin,  colmax 
scatter  matrix 

} 


Figure  22:  Region  data  structure 


6.1.1.  RegioB  OescriptioD 


Figure  27  describes  a  region  which  has  various  geometrical  properties  in  addition  to  statistical  properties. 


Texture.  Texture  value  extraction  algorithm  is  as  follows; 
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Struct  boundaiy{ 
boundary  identification  number 

vertex  identification  numbers 
region  identification  numbers 
number  of  lines 
line  identification  numbers 

length 

contrast 

chain  code  lengdi 
chain  code 

} 


Figure  23:  Boundary  data  structure 


struct  vatex{ 

votex  identification  number 

number  of  boundaries 
boundary  identification  numbers 

postion  row,  coluiim 

} 


Figure  24:  Vertex  data  structure 


1.  Derive  a  binary  edge  image  fiom  the  black  &  white  image  with  a  3  x  Laplacian  iterator  shown  in  Figure  28 
(a)  and  a  cutoff  value  T  (  7=  (standarddeviation)  x  1.4  ), 

2.  By  utilizing  the  9  x  9  window  shown  in  Figure  28  (b)  on  the  binary  picture,  calculate  the  number  of  ”1”$  in 
the  window.  The  number  is  the  texture  value  for  the  pixel  at  the  cento’  of  the  9x9  witKlow. 


Perimeter.  Perimeter  is  a  sum  (tf  the  arc  lengths  of  ail  boundaries,  counting  horizontal  and  vertical  moves  as  one 
and  diagonal  moves,  which  need  two  steps,  as  V2. 


Mass  Center.  Mass  center  of  a  region  is  computed  as  follows: 
struct  line{ 

line  idoitification  number 

boundary  identification  number 

start  row,  column 
end  row,  column 
distance  from  origin 
orientation 
length 

} 


Figure  25;  Line  data  structure 


74 


R:R*gloii 
B '.Boundary 
V:Vortox 
Li.lno 


Figure  26:  The  relation  desciiptitMi 


Figure  27:  The  region  description 
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where  N  is  the  number  of  pixels  in  the  region.  Pi  is  the  position  vector  of  i-th  pixel. 


Minimum  Bounding  Rectangle.  The  Minimum  Bounding  Rectangle  (MBR)  is  defined  as  a  set  of  four  numbers: 
rowmin,  rowmax,  colmin,  and  colmax.  They  are  the  maximum  and  minimum  values  of  the  pixel  locations  in  the 
region. 


Scatter  Matrix.  The  scatter  matrix  rqiresents  an  elliptical  area  which  approximates  the  slugte  of  the  region.  The 
scatter  matrix  C  of  the  region  can  be  calculated  as: 

t  ^ 

whoe  N  is  the  number  of  pixels  in  the  region,  Pi  denotes  the  position  vector  of  i-th  pixel,  and  M  denotes  the  mass 
center  of  the  region. 
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(b)  9  X  9  wndow  oparaior 

Figure  28:  A  L^lacian  operatm  and  9x9  window 
6.12.  Boundary  Description 

Hguie  29  describes  a  boundary.  Boundaries  are  the  borders  of  regions  in  a  region  m^.  They  are  not  related  to  the 
color  edge  detector. 


Column 


length 

contrast 


Figure  29:  The  boundary  segment  description 


Length.  Length  is  a  sum  of  the  arc  lengths  of  the  boundary,  counting  horizontal  and  vertical  moves  as  one  and 
diagonal  moves,  which  need  two  steps,  as  y/l. 


Contrast  Contrast  is  a  difference  of  mean  intensity  BlackA White  between  the  regions  forming  the  boundary. 
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6.U.  Vertex  Desariptioi 

Only  thfee  or  four  boundaries  can  meet  at  a  vertex  because  oi  the  square  tessellation. 

6.1.4.  Line  Description 

Hguie  30  describes  a  line.  Lines  are  obtained  by  applying  the  Miwa  line  extractor  [2S]  to  tte  edges  derived  by 
the  cokv  edge  detector  [20].  The  Miwa  line  extracted  has  a  pair  of  connected  arms  tracking  along  the  edges  and 
determines  a  line  by  checking  the  angle  formed  by  the  arms. 


Figure  30:  The  line  segment  description 


62.  Initial  Hypothesis  Generation 

The  initial  hypothesis  generalw  picks  up  some  tractaUe  regions  and  makes  initial  hypotheses.  It  is  skeptical  and 
tries  to  avoid  erroneous  hypotheses. 

Clancey’s  classificatkm  problem  solving  model  [26]  is  applied  to  initial  hypothesis  generation.  By  ana¬ 
lyzing  more  than  10  expert  systems  such  as  MYCIN.  SACXX4,  The  Drilling  Advistx’,  GRUNDY,  and  SOPHIE-in, 
he  discovered  a  generic  problem  solving  strategy  for  classification  problem  solving,  which  these  systems  have  in 
common.  Gassificatkm  problem  is  a  problem  \tiiich  classifies  input  data  into  known  solutions.  If  the  number  of 
solutions  is  fitirly  limited,  the  model  works  very  well.  It  has  three  stqM.  The  first  stq)  is  data  abstraction;  the  input 
data  are  abstracted  and  converted  to  more  tractable  forms.  The  second  stq>  is  heuristic  match;  data  abstractions 
are  projected  heuristically  into  solution  classes.  Each  solution  class  groups  similar  solutions.  The  last  step  is 
refinement;  one  of  the  known  solutions  in  that  solution  class  is  selected  by  refining  the  data. 

Figure  31  shows  how  the  classification  problem  solving  model  is  applied  to  the  initial  hypothesis  generation 
for  color  road  scene  interpretation.  The  intermediate  rqnesentation  data  are  first  abstracted  simple  rules  and 
the  system  makes  data  abstractions  such  as  large  area,  green  area,  lengthwise  area,  parallel  lines,  etc.  One  region 
gets  as  many  data  abstractions  as  possible.  Second,  some  regions  with  data  abstractions  are  projected  into  solution 
classes:  on  the  level  of  ground  or  above  ground.  On  the  level  of  ground  groups  roads,  grass,  soil,  shadows,  and  sun 
spots.  Above  ground  groups  tree  uimks,  tree  foliage,  vertical  object,  and  sky.  Last,  each  region,  which  has  fallen 
into  a  solution  class,  is  refined  and  labeled  with  a  solution  selected  out  of  the  members  of  its  solution  class. 

Solution  classes  are  determined  because  the  qjpearances  of  the  objects  protruding  from  the  ground  are  very 
different  firom  those  of  the  objects  on  ground  level.  When  tracing  objects  vertically,  the  appearance  of  the  above 
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Figure  31:  Classification  probtem  solving  nuxlel  applied  to  initial  hypothesis  generator 

ground  objects  such  as  tree  trunks  does  not  change  so  much,  whereas,  the  on  the  level  of  ground  objects  such 
as  roads  change  drastically  because  of  the  perepecdve  projection.  When  generating  initial  hypotheses,  it  is  less 
dangerous  to  pick  up  one  of  the  solution  classes  first  and  then  refine  it  than  to  pick  up  one  of  the  solutions  directly. 


6,2.1.  Data  Abstractioa 

All  the  data  abstractions  are  listed  in  Table  1.  There  ate  six  categories:  area,  location  (in  a  image),  color,  texture, 
shape,  and  line.  The  color  category  has  three  subcategories:  bright,  normal,  and  dark.  Data  abstraction  rules  are 
simple.  They  check  some  features  of  regions  and  goierate  data  abstractions.  Color  data  abstractions  are  not  very 
exclusive.  For  example,  one  region  can  be  blue  area  and  greener  area  at  the  same  time. 

Data  abstraction  is  the  most  important  component  of  the  initial  hypothesis  generation.  Thus,  it  is  one  of  the  most 
important  components  in  the  road  scene  intopreter.  Though  it  partly  depends  on  the  low-level  region  segmentation 
as  mentioned  in  Section  1.,  some  robust  ideas  are  embedded  in  the  data  abstraction  rules: 


Look  at  Neighbor  Regions.  Compare  features  with  those  of  the  neighbor  regions,  then  more  reliable  data  ab¬ 
stractions  can  be  generated.  Though  global  information  of  the  scene  is  not  available,  rules  can  check  the  neighbors 
locally.  Two  actual  examples  are  shown  in  Figure  32.  If  a  region  is  darker  than  its  neighbor  regions,  the  region  is 
really  dariL  If  the  coIot  of  a  region  is  closest  to  pure  green  (Red=0,  Green=2SS,  Blue=0)  among  neighbor  regions, 
it  is  guaranteed  that  the  region  is  really  green. 


Use  Color  Ratio  .  Use  the  color  ratio  such  as  Red/Blue  and  Gteen/Blue.  Do  not  use  R,  G,  and  B  intensities 
directly  except  for  detecting  dark  or  bright  regions.  R,  G,  arxl  B  intensities  vary  significantly,  depending  on  weather, 
camera  iris,  camera  white  balance,  video  amplifier,  A/D  converter,  etc.  R,  G,  and  B  intensities  were  used  directly  in 
the  ancestral  system  INSIGHT-n,  but  as  the  number  of  images  increased,  contradicting  intensities  were  obsmed. 
Consequently,  in  INSIGHT-m,  the  color  usage  was  totally  rebuilt  by  discarding  direct  color  intensity  refaences  and 
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Category 

Data  Abstracion 

Area 

Location 

higher  area,  lower  area 

Color 

Bright 

Normal 

Dark 

bright  area,  bright  red  area, 
bright  green  area 
blue  area,  bluest  area, 
green  area,  greener  area, 
brown  area 

dark  area,  darker  area, 
darkest  area 

Texture 

textural  area 

Shape 

lengthwise  area, 
crossw^area, 
vertical  parallel  area, 
trapezoidal  area 

Line 

long  line,  linear  lines, 
horizontal  line,  verticai  line, 
vertical  paraiy  lines, 
trapezoidal  lines 

Table  1:  Data  abstraction  rules 


introducing  color  ratios.  Examples  are  in  Hguie  33.  In  the  case  of  a  blue  area,  as  the  strength  of  blue  decreases, 
the  rule  checks  the  red  property  so  as  to  distinguish  blue  areas  from  brown  areas. 


Detect  Very  Dark  and  Very  Bright  Regions.  Detea  the  very  darit  and  very  bright  regions  beforehand  and  do 
not  use  their  color  infonnation,  because  their  cokxs  are  severely  distorted  owing  to  the  limited  dynamic  range  of 
the  camera.  The  two  solutions:  shadows  and  sun  spots,  ate  in  a  sense  for  tlKse  kinds  of  regions,  though  the  dark 
(H'  bright  regions  do  not  necessarily  have  to  be  labeled  with  shadows  ot  sun  spots.  Two  lands  of  data  abstractions: 
very  dark  and  very  bright,  are  requisite  fw  outdocv  natural  ct^or  scene  intopretation. 


6.2.2.  Heuristic  Match 

There  are  3  kinds  of  heuristic  match  rules:  on  the  level  of  ground,  above  ground,  and  both,  which  are  listed  in 
Table  2.  Most  of  them  ate  for  inferring  on  the  level  of  ground  objects.  The  reason  is  that  two  above  ground 
objects,  tree  foliage  and  sky,  ate  more  ambiguous  than  those  on  the  level  of  ground.  They  could  be  grass  and 
road,  respectively.  So,  they  usually  remain  unlabeled  in  the  initial  hypothesis  generation,  since  they  can  be  easily 
disambiguated  by  the  position  of  the  vanishing  point  of  the  toad  which  will  be  available  later.  One  rule,  which 
infen  an  above  ground  object  and  an  on  the  level  of  ground  objea  at  the  same  time,  is  intended  to  detea  pain  of 
a  tree  trunk  (vertical  object)  and  its  shadow.  It  turned  out  to  be  a  powerful  rule.  Some  examples  are  in  Figure  34. 


6.2J.  Rednement 

Each  solution,  grass,  soil,  shadow,  sun  spot,  tree  foliage,  or  sky  has  one  region-based  refinement  rule  which  checks 
the  solution  class  and  other  data  abstractions.  Road,  tree  trunk,  or  vertical  object  has  a  region-based  refinement 
rule  and  an  edge-based  refinement  rule,  since  the  shapes  of  these  objects  ate  much  mote  constrained  than  the  other 
objects.  Figure  35  explains  some  rules. 
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Darker  Area: 

IF  biaci<&whitB<64 
and  (20<-b<ack&wriitB  or  20<-red ) 
and  green<-biue432 
and  daikest  among  neighbor  regions 
THEN  The  region  is  ’darker  area” 


Greener  Area: 

IF  48<green 
and  biue4-i0<green 
and  closest  to  pure  green(0, 255,0) 
among  n^bor  regions 
THEN  The  region  is  ’greener  area* 

Figure  32:  I>ata  abstraction  niles  by  looking  neighbor  regions 


Bluest  Area: 

IF  48<d)lue  and  g/b<-0.90 
THEN  The  region  is  'bluest  area’ 


Blue  Area: 

IF  36<dQlue  and 
[  gto«-0.97 

or  (g/tx*1.05  and  r/b<»1.40) 
or  (g/bc-i.lO  and  r/b<»125)  ] 
THEN  The  region  is  ’blue  area’ 


g/bsgreen/blue,  r/b-red/blue 


Figure  33:  Data  abstraction  rules  by  using  cdor  ratio 


63.  Context  Control 

Figure  36  illustrates  the  context  control  of  the  road  scene  intopreter.  It  looks  slightly  different  from  that  in  Figure  4, 
though  they  are  basically  the  same.  After  the  extrapolation,  the  interpretation  cycle  goes  back  to  the  heuristic  match 
of  the  initial  hypothesis  generate.  The  reason  is  that  a  part  of  the  t^  of  the  extrapolator  is  very  similar  to  that  of 
the  heuristic  match  and  refinement.  Simply,  the  extrapolator  shares  some  rules  with  the  initial  hypothesis  generator. 


6.4.  Evaluation 

Each  labeled  region  has  a  plausibility  value.  In  each  intrapretation  cycle,  labeled  regions  are  evaluated  and  get  their 
plausibility  values  increased  or  decreased.  If  a  plausibility  value  becomes  negative,  its  label  is  rmoved  and  gives 
chatKes  to  alternative  labels.  Each  region  has  2  slots  fw  memorizing  2  previous  labels,  so  that  the  removed  label 
cannot  take  place  again  unless  2  alternative  labels  have  been  tried. 

Thble  3  shows  the  composition  of  the  evaluation  rules.  Evaluation  rules  are  categorized  into  local  and  global 
evaluation  rules.  Local  and  global  evaluation  rules  have  positive  and  negative  evaluation  rules.  A  local  evaluation 
rule  picks  up  a  pair  of  labeled  regions  at  a  time  and  checks  their  consistency.  A  global  evaluation  rule  picks  up 


Data  Abstraction  Heuristic  Match 


higher-da»i«er-lengthwise.  Above  ground 
parallel  area 


large-lower, 

large-blue, 

bluest, 

bright-lower, 

large-darkest-crosswise, 

tra^oidal, 

moderatelyjarge-lower 

-green-touching 

vertical  frame 


On  the  level 
of  ground 


Above  ground 
and  On  the 
level  of  ground 


Table  2:  Heuristic  match  rules 


Above  Ground: 

IF  no  solution  class  is  generatad 
and  higher  area 

and  (darkar  area  or  darkest  area) 
and  lengthwise  area 
THEN  It  is  "above  grouncT. 


On  the  Level  of  Ground  Region: 

IF  no  solution  class  is  generated 
and  large  area 
and  lower  area 

THEN  It  is  "on  the  level  of  ground”. 
Figure  34:  Heuristic  match  rule  examples 


one  labeled  region  at  a  time  and  checks  its  consistency  with  one  of  the  explicit  scene  models.  Four  of  the  negative 
global  evaluation  rules  are  dominant  When  the  plausibility  values  of  explicit  scene  models  become  reliable,  these 
four  rules  enforce  the  model  by  rejecting  labeled  regions  contradicting  the  models. 

Some  local  evaluation  rules  are  in  Figure  37.  If  a  pair  of  labeled  regions  is  consistent  the  rule  gives  S  points 
to  each  labeled  regions.  If  a  pair  of  labeled  regions  is  inconsistent  the  rule  lakes  8  points  firom  the  less  plausible 
labeled  region  and  2  points  firom  the  mme  plausible  region;  if  they  are  equally  plausible,  it  takes  S  points  from 
the  each  labeled  region.  When  a  pair  of  labeled  regions  ate  inconsistent  one  U^led  region  could  be  good  and 
the  other  one  could  be  bad.  The  negative  evaluation  rule  can  not  figure  out  locally  which  one  is  wrong.  But  the 
good  labeled  region  must  survive.  However,  each  labeled  region  has  a  plausibility  value.  If  the  plausibility  value  is 
high,  this  means  the  region  has  survived  a  couple  of  interpietatkm  cycles  and  got  its  plausibility  value  increased  by 
positive  evaluation  rules.  Thus,  the  mote  plausible  labeled  region  should  have  an  advantage  over  the  less  plausible 
labeled  region. 

Some  global  evaluation  rules  are  in  Figure  38.  After  a  couple  of  interpretation  cycles,  the  plausibility  of  the 
models  becomes  moderately  high,  and  then  the  global  evaluation  starts  in  addition  to  the  local  evaluation.  If  a 
labeled  region  is  consistent  with  a  related  model,  the  rule  gives  10  points  to  the  labeled  region.  If  a  labeled  region 
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Soil  Found: 

IF  on  the  level  of  ground 
and  txown  area 
and  not  darkest 
and  32<MBR_rowmax 
and  road  model  plausSiilityousable 
THEN  The  region  is 


Road  Trapezoid  Found: 

IF  on  the  levai  of  ground 
and  bhje  area 
and  trapezoidal  area 
and  not  textural  area 
and  road  model  plausibilityousable 
THEN  The  region  is  "roatf. 


Figure  35:  Refinement  nde  examples 


Category 

Positive 

Negative 

Totai 

Local 

7 

11 

18 

Ordinary 

4 

7 

11 

pominant 

- 

■1 

Bi 

Total 

11 

22 

33  1 

Table  3:  Evaluation  rules 


is  inconsistent  with  a  related  model,  the  rule  takes  10  points  Grom  the  labeled  legioiL  When  the  plausibility  of  the 
models  becomes  reliable,  four  dominant  negative  global  evaluation  rules  can  fire.  If  a  labeled  region  is  inconsistent 
with  a  related  model,  the  dominant  negative  global  evaluation  rule  takes  10000  points  from  the  labeled  region,  so 
that  the  label  is  removed. 


6S.  Modeling 

Initially,  one  road  model  is  already  generated  and  ready  to  be  adjusted  as  dqiicted  in  Figure  2  (a).  The  model  is 
adjusted  to  the  latest  interpretatum  in  each  cycle  by  the  functions  written  in  C-h-.  In  each  model  adjustment,  first, 
the  vanishing  point  is  moved  horizontally  until  the  discrepancy  reaches  a  local  minimum;  then,  the  lower  end  of 
the  left  road  edge  is  slid  horizontally  until  the  discrqtancy  reaches  a  local  minimum;  then,  the  lower  end  of  the 
right  rood  edge  is  slid  horizontally;  last,  the  vanishing  point  is  moved  vertically.  This  road  model  adjustmoit  cycle 
is  iterated  rnie  to  three  times  (landing  on  the  idausibility  level  of  the  road  model. 

Model  adjustment  for  each  degree  of  freedom  is  not  very  expensive.  Only  the  pixels  along  the  boundary  need 
to  be  tested.  Rgure  39  explains  it.  In  the  figure,  the  lower  etid  of  the  left  edge  is  being  adjusted.  If  it  moves 
left,  the  area  shaded  with  dots  minus  the  area  shaded  with  grids  improves  the  discrepancy  value.  If  it  moves  right, 
the  discrepancy  value  deteriorates.  Thus,  it  goes  left.  Then,  it  calculates  the  gain  area  minus  the  loss  area  for 
the  next  step.  It  keqts  going  left  as  long  as  the  calculated  value  is  positive.  Whoi  the  calculated  value  becomes 
negative,  it  stops  and  its  position  is  the  local  minimum  of  the  discrepancy  value.  The  same  operation  is  applied  to 
the  other  degrees  of  fteedom  one  after  another  until  the  discrq)ancy  value  settles.  The  required  computing  power 


82 


Figure  36:  Context  ctMilrol 


is  proportional  to  the  number  of  degrees  of  freedom,  since  only  one  degree  of  freedom  is  adjusted  at  a  time. 

Modeling  is  one  of  the  most  important  compmients  in  the  road  scene  inteirpreter.  The  explicit  road  scene 
models  extract  global  information  the  road  scene,  on  which  the  evaluator  and  the  extrapolator  rely.  If  the  global 
information  of  the  models  is  wrong,  the  whole  interpretation  is  confused  and  cornqrted.  Especially  for  the  first  one 
or  two  interpretation  cycles,  the  models  are  unstable  simply  because  there  are  only  several  labeled  regions.  So,  for 
the  first  a  few  interpretation  cycles,  the  models  must  be  less  sensitive  until  quite  a  few  regions  are  labeled.  We 
observed  that  the  models  were  once  oversensitive  and  adjusted  very  strangely.  A  couple  of  ideas  are  adopted  in 
order  to  make  the  models  less  soisitive  for  the  first  a  few  interpretatiai  cycles: 


Do  Not  Adjust  a  Road  Model  Edge  Unless  Grass  or  SoQ  is  Found  on  that  Side.  Grass  or  soil  regions  prevent 
the  road  model  from  expanding.  If  there  is  no  grass  or  soil  region  found  on  the  right  or  left,  the  lower  end  of  the 
right  or  left  road  edge  of  the  model  can  expand  too  much.  So,  if  no  grass  or  soil  is  found,  the  lower  end  of  the 
road  edge  of  the  model  on  that  side  should  not  be  adjusted. 


Restrict  the  Vanishing  Point  and  Width  of  the  Road  Model  Loosely.  It  is  very  strange  if  the  vanishing  point 
moves  far  off  the  top  edge  of  the  image  or  very  close  to  the  bottom  edge  of  the  image.  If  the  road  width  becomes 
twice  as  wide  as  expected,  this  is  also  unusual.  These  situations  should  be  avoided  for  the  first  few  interpretation 
cycles. 

Though  it  belongs  to  domain  knowledge  organization,  experimentally,  a  Y-intersection  detection  mechanism  has 
been  implemented  to  confirm  the  extendibility  of  the  adjust^le  explicit  scene  models  and  the  interpretation  cycle. 
Figure  40  shows  its  basic  idea.  Since  the  road  adjustment  looks  for  a  local  minimum  of  the  discrqrancy  value,  a 
road  model  is  adjusted  to  one  of  the  roads  by  chance.  The  other  toad  makes  the  discrepancy  value  big.  So,  if  there 
is  big  discrepancy  on  either  right  or  left,  the  modeler  generates  an  additional  road  model  in  that  directim.  Figure  21 
is  an  actual  result  The  modeler  is  also  capable  of  eliminating  erroneous  additional  road  models.  Elimination  of 
an  erroneous  additional  road  model  is  easy,  since  the  vanishing  pewt  of  the  toad  model  becomes  very  close  to  the 
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Local  Positiva  Evaluation: 

IF  a  foliage  and  a  tree  trunk 
are  ac|aoent 

and  foliage's  mass  center  is 

heigher  than  that  of  the  toink 
THEN  give  each  region  5  points. 

Local  Negative  Evaluation: 

IF  a  road  and  a  tree  trunk 
are  ac|aoent 

and  the  trunk  is  the  highest 

among  ac|acent  trunks 
and  top  of  the  road  is  higher 
than  top  of  the  tnjnk 
and  usable<«road  model  plausibility 
THEN  take  8  points 

from  less  plausible  region 
and  taka  2  points 

from  rrtote  plausbie  region 
when  they  are  equally  plausible 

taka  5  points  from  each  region 


Figure  37:  Local  evaluation  rule  examples 


main  road  model  if  it  is  inappropriate. 

Adjustment  of  the  tree  trunk  model  is  one  degree  of  freedom  after  another,  mass  center  row,  mass  center 
column,  width,  height,  and  lean.  Each  degree  of  freedom  finds  a  local  minimum  of  the  discrepancy  value  and  this 
model  adjustment  cycle  takes  place  once  in  each  interpretation  cycle.  Generation  and  elimination  of  the  tree  trunk 
models  differ  from  those  of  the  road  models.  Every  time  a  fragment  of  a  tree  trunk  is  found,  a  tree  trunk  model 
IS  generated  and  adjusted  independently.  Thus,  one  tree  trunk  may  have  several  tree  trunk  models  at  first.  By 
checking  the  overlaps  of  the  tree  trunk  models,  some  models  are  eliminated  and  one  model  remains  for  each  tree 
trunk.  This  is  shown  in  Figure  20  (h).  If  a  tree  trunk  model  becomes  very  small,  it  is  also  eliminated. 

As  the  interpretation  progresses,  the  plausibility  level  of  the  models  becomes  higher.  Table  4  shows  the  five 
plausibility  levels  of  the  road  model.  It  is  determined  by  the  area  of  the  reliable  regions  whose  plausibility  value 
is  ttreater  than  or  equal  to  10.  At  first,  the  plausibility  level  of  the  models  is  not  usable  since  the  models  are  not 
reliable.  Then,  it  becomes  carefully  usable;  the  system  <iscs  the  models  but  still  checks  the  features  of  regions 
carefully.  Then,  it  becomes  usable;  the  system  uses  the  nodels  and  checks  Just  a  few  features  of  regions.  Then, 
it  becomes  reliable;  the  system  uses  the  models  without  checking  the  features  of  the  models.  LasL  it  becomes 
dominant;  the  system  totally  relies  on  the  model  and  rejects  all  of  the  inconsistently  labeled  regions.  It  can  use 
the  object  label  unknown  in  this  plausibility  level  and  quits  interpretation  as  soon  as  possible.  Currently,  only  the 
plausibility  level  of  the  main  road  model  is  calculated  and  represents  all  the  plausibilities  of  the  models. 


6A.  Extrapolation 

Labeled  regions,  which  have  survived  some  interpretation  cycles,  propagate  to  their  adjacent  unlabeled  regions  by 
checking  the  location,  shape,  color,  area,  and  so  on.  Compared  to  initial  hypothesis  generation  rules,  less  strict 
conditions  are  required  in  extrapolation  since  each  rule  is  based  on  one  labeled  region.  Extrapolation  rules  are 
simple  and  make  easy  guesses.  Most  of  the  guesses  will  be  good,  but  some  of  them  might  be  wrong.  They  are 
Judged  by  the  evaluator.  Table  5  lists  the  categories  of  the  extrapolation  rules. 

1/3  of  the  extrapolation  rules  infer  information  aboc*  solution  classes.  Two  of  them  remove  on  the  level  of 


84 


Global  Poaltiva  Evaluation: 

IF  a  tree  trunk  is  inside  a  tree  model 
and  carefully-usable<amodel  plausibility 
and  it  is  dark  area,  darker  area, 
or  darkest  area 

and  it  is  not  on  the  level  of  ground 
THEN  give  the  region  10  points. 


Global  Dominant  Negative  Evaluation: 

IF  a  tree  tnrnk  is  not  inside 
any  tree  tnmk  model 
and  reliable<smodel  plausibility 
THEN  take  10000  points 
from  the  region 


Figure  38:  Global  evaluation  rule  examples 

The  dbection  of  the  movement  can  be  delennined 
by  checking  only  the  labeled  pixels  along  the 
bourtdary. 

In  the  Wustratlon,  Vthe  bottom  of  the  Ml  edge 
moves  left,  the  discrepancy  value  Is  Improved. 


Bi  Gam  w  Loss 


Figure  39:  Model  adjustment  algorithm 

ground  solution  class  or  above  ground  solution  class.  Removal  of  a  solution  class  assigned  is  often  caused  by  the 
transition  of  the  vanishing  point  of  the  road  model.  The  rest  of  the  rules  generate  solution  classes.  Figure  41  shows 
an  example. 

Two-thirds  of  the  extrapolation  rules  infer  infcHmadon  about  solutions.  Adjacency  is  a  primary  cue  for  extrapo¬ 
lation  and  used  by  most  of  the  rules.  For  instance,  a  tree  trunk  expects  tree  foliage  above  it  Figure  41  has  anoAer 
example.  A  road  fragment  will  be  adjacent  to  anodier  road  firagment  Occasionally,  a  small  spot  such  as  a  crack  on 
a  road  or  a  dent  in  a  tree  trunk  forms  an  independent  small  region  surrounded  by  similar  large  regions.  This  kind 
of  small  regioiu  are  deduced  if  the  surrounding  regions  take  the  same  label.  Two  rules  use  the  sun  direction  su.. 
and  try  to  propagate  tree  trunk  regions  to  that  direction,  since  the  sunny  side  of  the  tree  trunk  is  often  laight  and 
ambiguous.  An  actual  example  is  shown  in  Figure  20  (e)  (g),  where  the  tree  trunk  expands  to  the  right  Two-fifth 
of  the  rules  use  scene  models  to  infer. 

One  weather  rule  determines  whether  it  is  sunny  or  not  by  simply  checking  the  existence  of  sun  spots  or 
shadows.  Then  another  rule  detects  the  direction  of  the  sun  by  finding  an  appropriate  pair  of  a  shadow  and  a 
vertical  object  touching  at  the  bottom  of  the  vertical  object  Unless  shadows  ate  vertical  in  the  image,  it  is  fairly 
easy  to  detect  the  direction  of  sun  and  utilize  it  in  interpretation. 
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If  then  is  big  dtaaepancy  on  either  left  or  right, 
the  system  generates  additional  road  model. 


If  the  wnWiing  poM  of  the  addUon^'  road  model 
becomes  ve^  doee  to  the  main  road  model,  the 
s^tem  eliminates  the  addftional  road  model. 


Figure  40:  Intersection  model 


6.7.  Road  Map  Generation 


Because  explicit  road  scene  models  extract  the  vanishing  point  of  a  road  and  the  bottoms  of  trees,  a  road  map  can 
be  generated  under  an  assumption  that  the  roots  of  trees  are  on  the  level  of  ground.  If  its  view  frame  angle  ^  is 
known,  then  inverse  projection  becomes  possible. 

Figure  42  illustrates  the  inverse  projection.  The  projection  plane  (image  plane)  is  set  on  the  lower  base  of  the 
trapezoidal  view  frame.  All  we  have  to  know  is  the  location  of  the  center  of  projection  and  the  tilt  angle  of  the 
projection  plane.  If  the  angle  9  of  the  view  frame  is  givot,  by  extending  the  sides  of  the  view  frame,  the  locaticm  of 
the  center  of  projection  in  the  plane  view  is  derived  as  defucted  in  Figure  42.  The  distance  d  between  the  projection 
plane  and  the  center  of  projection  in  the  plane  view  is  c^culated  as 


d  = 


-cot-, 
2  2 


where  is  the  length  of  the  lower  (shorter)  base  trf  the  trapezoidal  view  frame  as  in  the  plane  view  in  Figure  42. 
Note  that  if  we  draw  a  perpendicular  line  segment  from  the  center  of  projection  to  the  projection  plane,  the  foot  of 
this  segment  lies  at  the  center  of  the  image.  Now  the  remaining  unknowns  are  the  height  of  the  center  of  projection 
and  the  tilt  angle  (tf  the  projection  plane.  Let  us  think  of  the  plane  determined  by  the  center  of  projection  and  the 
vanishing  point  level  in  the  image  plane.  This  plane  must  be  parallel  to  the  ground  plane.  By  tilting  the  projection 
plane  until  this  plane  becomes  parallel  to  the  ground  plane,  the  height  of  the  center  of  projection  and  the  tilt  angle 
are  determined  at  the  same  time  as  drawn  in  the  side  view  in  Figure  42. 


6.8.  System  Analysis 


Table  6  shews  the  composition  of  rules.  The  initial  hypothesis  generator  consists  of  60  rules:  data  abstraction  27, 
heuristic  match  11,  and  relinement  22.  This  is  the  largest  rule  module.  The  extiapolator  has  52  rules  and  is  the 
second  largest  rule  module.  The  modeler  has  only  16  rules,  because  most  of  it  is  written  in  C-h-  fimetions.  If  C-h- 
functions  are  taken  into  treount,  the  modeler  is  the  biggest  module.  In  total,  the  intnpreter  comprises  187  rules  to 
date.  The  source  code  of  the  interpreter  has  about  8300  lines,  2/3  in  OPS83  and  1/3  in  C-m-.  If  the  source  code 
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The  road  model  has  5  levels  of 
plausibility. 

Redo  of  reliable 
regions  In  a  scene 
(10<reglon  plausibllily  value) 

1.  Unreliable 

2.  Carefully>usable  1/3 

3.  Usable  1/2 

I  usually  1  interpretation  cycle 

4.  Reliable  1/2 

I  2  Interpretation  cycles 

5.  Dominant 

(quit  interpretation  as  soon  as  possfcle) 


Table  4;  Plausibility  levels  of  the  model 


of  the  feature  extraction  program  is  added,  it  exceeds  10000  lines.  Since  main  body  of  the  interpreter  is  written  in 
rule-based  programming  language  OPS83,  it  is  still  very  maintainable. 

There  are  two  important  modules:  data  abstraction  and  modeling.  The  data  abstraction  of  initial  hypothesis 
genoation  is  the  first  knowledge  base  applied  to  the  intermediate  data.  The  very  first  stage  always  plays  an 
influential  role  in  a  total  system.  It  connects  the  intermediate  level  and  the  high  level,  and  preserves  its  pmallelism 
because  rules  in  the  data  attraction  are  independent.  Modeling  is  significant  because  adjust^  scene  models  become 
influential  and  then  dominant  in  the  latto-  half  of  the  interpretation.  Once  the  models  start  converging,  evraything 
goes  well.  Evoy  effort  in  knowledge  acquisition  is  made  such  that  the  scene  models  converge.  When  loddng  at  a 
new  road  image,  the  first  task  of  the  knowledge  engineer  is  to  mentally  fit  a  triangular  road  model  to  the  road.  No 
matta  how  much  the  toad  is  shaded,  if  the  road  model  can  be  adjusted  properly,  it  is  worth  working  on  the  image. 

Negative  evaluation  is  not  troublesome  as  oppose  to  the  intuition.  Actually,  evaluation  is  the  smallest  one  among 
four  modules  as  it  is  in  Table  6  (modeler  is  the  largest  since  its  fimctions  are  m  C-H-).  3  reasons  are  enumoated: 


1.  Global  negative  evaluation  is  reliable  as  long  as  models  are  ptopaly  adjusted. 

2.  Since  local  negative  evaluation  cannot  avoid  mistakes,  the  simple  strategy  in  Section  6.4.  is  not  much  different 
from  the  optimal  strategy. 

3.  The  basic  interpretation  strategy,  cut-and-try,  of  the  system  with  interpretation  cycles  alleviates  the  evaluation 
task. 


Four  modules,  initial  hypothesis  generation,  evaluation,  modeling,  and  extr^lation,  are  in  a  certain  balance. 
None  of  the  modules  is  perfect  The  four  modules  work  together  to  compensate  for  each  other  and  poform 
reasonable  interpretation.  So,  when  modifying  the  syst^,  the  balance  needs  to  be  taken  into  account  The 
knowledge  engineer  has  to  recognize  the  weakest  module  and  enhance  it  Another  aspect  of  the  system  is  that  one 
rule  could  be  fired  mcve  than  SO  times  because  of  the  cyclic  mechanism.  For  example,  there  can  be  more  than 
20  pairs  of  road  and  grass  regions  in  a  scene.  A  road-grass  positive  evaluation  rule  is  fired  on  each  pair  of  these 
regions  in  every  interpretation  cycle  in  order  to  keep  them  alive.  This  characteristic  is  very  different  from  oflier 
production  systems. 

Since  the  system  adopted  a  cyclic  mechanism,  various  phenomena  such  as  oscillation,  overshoot,  divergence,  etc. 
studied  in  control  theory  can  potentially  happen.  Those  phenomena  were  observed  while  developing  INSIGHT-m. 
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Category 

Number 
of  rules 

Solution 

Class 

higher/lower  than 
vanishing  level 

6 

higher  than  a 
above-ground  region 
lower  than  a  on-the 
-level-of-ground  region 

2 

similar  to 

3 

surrounded  by 

1 

with  sun  direction 

2 

Solution 

object  A  expects 
object  B 

9 

similar  to 

6 

surrounded  by 

5 

with  sun  direction 

2 

by  scene  model 

13 

1  Total 

49 

Table  S:  Extrapolation  rules 


7.  Future  Work 

To  enhance  the  initial  hypothesis  generator,  a  couple  effective  improvemrats  are  possible  and  discussed. 

Intermediate  representation  data  need  to  be  appropriately  abstracted  in  order  to  provide  reliable  initial  hypotheses 
for  the  following  interpretation  stages.  Convosely,  it  is  also  true  that  perfect  data  abstraction  cannot  be  generated 
locally.  However,  at  least  a  few  significant  regions,  in  terms  perceptual  organization  [27][28][29],  must  be 
properly  abstracted  in  a  bottom-up  manna:  as  initial  clues,  otherwise  there  is  no  way  to  interpret  a  natural  color 
scene  without  predictions.  Even  for  a  human,  if  a  scene  is  very  dark  and  does  not  give  good  initial  clues,  it  is 
difficult  to  interpret  Therefore,  pursuing  reliable  data  abstraction  in  a  bottom-up  manno'  is  at  least  a  necessary 
approach  to  more  robust  naunal  color  scene  interpretation. 


7.1.  inexhaustive  Region  Segmentation  for  High-level  Interpreter 


In  inexhaustive  region  segmentation,  some  ambiguous  pixels  are  not  necessarily  segmented.  It  alleviates  the 
segmentation  task  and  is  also  appropriate  to  the  high-level  interpreter. 

From  the  view  point  of  the  high-level  scene  interpreter,  little  consideration  has  been  given  to  low-level  region 
segmentatioa  The  objective  of  low-level  region  segmentation  has  been  roughly  how  well  an  image  can  be  segmented 
in  comparison  with  an  ideal  one  segmented  by  a  human.  This  objective  automatically  includes  semantics,  which 
is  not  very  successful  [30].  But  an  ideal  region  segmentation  for  a  high-level  interpreto*  can  be  differenL  The 
objective  of  low-levd  region  segmentation  should  be  to  give  pure  low-level  cues  to  a  high-level  interpreter.  In  the 
case  of  the  system  INSIGHT-m,  the  objective  must  be  to  give  pure  low-level  cues  to  the  data  abstraction  module 
of  the  initial  hypothesis  generate^'. 

C(dor  transition  occurs  near  the  boundary  of  two  regions.  But  those  transition  pixels  are  always  merged  into  one 
of  the  regions.  We  observed  that  the  average  colors  of  tte  segmented  regions  differ  from  those  of  the  corresponding 
portions  of  die  original  image.  This  is  caused  by  those  heterogeneous  pixels  along  the  boundaries  and  in  the  portions 
of  a  region  undersegmented.  If  heterogeneous  pixels  are  mixed,  the  averaged  color  tends  to  be  neutral.  This  color 
contamination  weakens  the  colcv  cues  and  mato  data  abstraction  erroneous.  Since  region  boundaries  are  not  very 
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Lower  than  vanishing  level  la  ground: 

IF  a  r^ion  has  no  solution  class 
and  vanishing  point  level  +  80 

<MBR_rowmax 
and  usable  <=  road  model  plausibility 

THEN  the  region  could  be  on  the  level 
of  ground 

Road  expects  grass: 

IF  an  unlabeled  region  is  adjacent  to 
a  road  region 

and  it  is  (green  area  or  bright  green  area) 

and  itis  not  lengtftwise  area 

and  road  model  plausibility  <a  usable 

THEN  the  region  could  be  grass. 


Figure  41:  Extrapolation  rule  examples 


important  to  a  high-level  interpreter  with  adjustable  explicit  scene  models,  exhaustive  region  segmentation  is  not 
requisite.  Initial  hypotheses  in  INSIGHT-m  are  often  dispersed  fragments  of  the  labeled  regions  as  in  Figure  19  (a), 
but  the  scene  models  are  adjusted  appropriately.  Therefore,  the  objective  of  the  region  segmentation  should  be  to 
extract  very  homogeneous  portions  of  the  image  for  reliable  data  attraction.  A  shaded  road  scene  image  contains 
many  ambiguous  pixels.  Those  ambiguous  pixels  along  region  boundaries  need  not  be  merged  to  the  regions. 

Inexhaustive  region  segmentation  gives  rise  to  some  discussion.  First,  what  is  the  image  interpretation  result, 
labeied  regions  or  adjusted  modeis?  It  depends  on  the  application.  For  autonomous  vehicle  navigation,  adjusted 
explicit  models  or  a  road  map  is  better.  If  the  global  information  extracted  from  inexhaustively  segmented  regions 
is  sufficienL  exhaustive  region  segmentation  is  no  longer  necessary.  Conversely,  some  application  may  need  labeled 
regions,  hi  this  case,  the  image  can  be  exhaustively  segmented  afterwards  with  the  global  information  extracted. 
Second,  can  explicit  scene  models  be  adjusted  appropriately?  Basically,  there  is  a  local  minimum  of  the  discrepancy 
value  near  the  best  match.  Although  the  gtqs  between  regions  may  make  some  weak  local  minima,  they  can  be 
overcome  by  modifying  the  adjustment  algtmthm  slightly.  This  will  be  much  easier  than  unlabeling  misinterpreted 
regions  caused  by  color  contamination.  Third,  how  can  relations  among  regions  be  expressed?  Adjacency  can  be 
defined  among  unconnected  regions.  For  example,  if  two  regions  are  within  a  certain  distance  at  the  closest  points 
and  no  otho’  region  is  between  them,  then  those  regions  are  adjacent  Last,  won't  it  lose  edge  ittformation?  Edges 
are  separately  extracted  with  color  edge  detector  as  depicted  in  Figure  6. 


72.  Adaptive  Data  Abstraction 


Since  the  interpret^'  is  capable  of  recognizing  interpretation  fiiilure  by  checking  the  consistency  of  adjusted  scene 
models  or  detecting  infinite  iteration  of  interpretation  cycles,  the  parameters  for  data  abstraction  can  be  changed 
several  times  until  the  interpreter  generates  reasonably  adjusted  scene  models.  In  this  case,  INSIGHT-m  is  used 
as  a  part  of  the  bigger  system  with  cycles.  This  meta-interpretation  cycle  works  as  a  mechanism  for  recovering 
from  fatal  interpretation  failure.  If  the  number  of  parameters  are  limited,  it  becomes  feasible.  While  develq)ing 
INSIGHT-m,  many  efforts  were  made  on  differentiating  road  and  soil  regions  and  it  was  of  importance  for  road 
extraction.  Often  these  two  regions  are  similar.  Both  are  on  the  level  of  ground  and  have  little  texture.  The  only 
cue  is  the  slight  difference  of  the  colors:  soil  is  a  little  tedder  and  darker  than  toad,  though  it  is  not  always  so 
because  of  camera  condition,  color  contamiruition,  noise,  etc.  This  difficulty  will  be  alleviated  by  this  adaptive  data 
abstraction. 
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Plane  view  Side  view 


Ccntw  of 
Q«tjnd  projoetion 

Vlow  tramo  (•vpl  {•ypi 


Road  scene  model 
(inverted) 


Figure  42:  Inverse  projection 


8.  Conclttsioa 

The  outdoor  natural  color  road  scene  interpreter  with  adjustaUe  explicit  scene  models  and  an  interpretation  cycle 
has  been  developed,  and  more  than  20  natural  road  scenes,  some  destructively  shaded,  are  reliably  int»preted.  By 
moderately  simplifying  the  low-level  region  segmentation  and  domain  knowledge  organization,  and  mtentionally 
selecting  ill-structured  scenes  instead,  some  of  the  issues  of  robust  natural  color  scene  inter^tadon  have  been 
clarified: 

1.  Global  information  of  a  natural  scene,  which  makes  the  interpretation  substantially  reliable,  can  be  extracted 
by  adjustable  explicit  scene  models  and  using  an  interpretation  cycle. 

2.  Negative  evaluation  is  not  troublesome.  It  performs  well  in  concert  with  global  information  extractiem. 

3.  Classification  problem  solving  model  is  appropriate  as  an  initial  hypothesis  generator  in  natural  scene  inter¬ 
pretation. 

4.  Color  contamination  caused  by  the  heterogeneous  pixels  along  boundaries  and  undersegmented  portion  of  a 
region  weakens  initial  hypothesis  generation. 
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Numbar  a(  rulea 


Context  control 

22 

Data  Abstraction  ^ 

27 

InWal 

Heuristic  match 

HypothMis  1 1 

Qeiwralion 

Refinement  ^ 

22, 

Evaluation 

33 

Modeling 

16 

Extrapolation 

52 

Map  generation 

4 

Total 

187 

Table  6:  Tbe  composition  of  rules 


A  The  Color  Median  Filter 

Abstract 

In  this  appendix,  the  mathematical  concept  median  is  extended  to  multiple  dimensions.  This  extended  concept 
vector  median,  is  then  applied  to  color  image  noise  redaction.  The  evaluation  of  this  color  median  filter  shows 
that  it  preserves  the  edges  of  a  color  image  more  than  the  method  which  smoothes  each  component  of  the  image 
by  median  filter  separately. 


Al.  Median  Filter 

The  median  filto’  [31]  is  a  powerful  smoothing  technique  that  does  not  blur  or  enhance  edges.  It  is  probably  the 
best  smoothing  filter,  when  the  characteristics  of  the  noise  are  unknown.  This  advantage  comes  from  the  fact  that 
the  median  filter  never  calculates  an  average;  in  other  words,  it  never  introduces  a  new  pixel  It  merely  replaces 
the  value  at  a  point  by  the  median  of  the  values  in  a  neighborhood  (rf  the  point. 

How  can  a  median  filter  be  applied  to  a  multi-band  image  such  as  a  coIot  image,  whose  pixels  are  vectors?  This 
had  been  in  question  since  median  was  defined  only  on  scalar  quantities.  To  inhoit  the  advantage  of  the  median 
filter,  a  new  pixel  should  not  be  introduced,  because  the  new  pbiel  could  blur  the  edge.  If  the  median  can  be  defined 
on  vector  quantities,  the  answer  is  straightforward:  the  median  filter  merely  replaces  the  vector  at  a  point  by  the 
median  vector  of  the  vectors  in  a  neighborhood  of  the  point  The  definition  is  discussed  in  section  A2. 

The  median  filter  had  been  used  unsatisfactrxily  on  the  red,  green,  and  blue  components  of  the  color  image 
separately.  This  method  introduces  new  pixels  whose  components  are  extracted  firom  the  different  neighbor  pixels. 
The  new  pixels  rtot  only  blur  the  edges  but  also  could  change  the  colors  disparately. 

An  extreme  case  is  depicted  in  Figure  43.  When  scanning  a  color  image  with  a  small  cross  window,  five  inxels 
could  be  at  magenta  (re±255,  greenrO,  blue:2SS),  magenta  (255,  0,  255),  cyan  (0, 255, 255),  green  (0,  255, 0),  and 
yellow  (255,  255,  0),  which  are  shown  in  color  space.  If  median  filter  is  utilized  on  each  component  separately,  it 
introduces  white  (255, 255, 255),  since  two  pixels  are  at  0  and  the  remaining  three  pixels  are  at  255  when  projected 
on  each  ted,  green,  or  blue  axis.  The  white  is  far  from  the  (Higinal  pixel  colors.  This  is  worse  than  averaging  filter 
that  will  introduce  gray  (153, 153, 153).  If  a  new  pixel  should  not  be  introduced,  which  one  of  the  five  pixels  is  the 
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most  appropriate  as  a  median?  Intuitively,  it  will  be  magenta,  because  there  are  two  pixels  there  while  the  others 
are  alone.  This  magenta  pixel  presaves  the  edge  more  than  the  white  (2SS,  255, 255)  or  the  gray  (1S3,  153,  153). 


Figure  43:  A  new  pixel  introduction 

If  median  filta  is  applied  to  each  component  separately,  the  extracted  vector  is  not  unique;  it  depends  on  the 
axes  chosen.  Figure  44  illustrates  this  dependency  in  two  dimensions.  The  axes  X  and  Y  select  the  vector  m  while 
the  axes  X'  and  Y  draw  the  vector  m^  The  extracted  vector  can  be  at  various  locations  when  the  axes  rotate.  This 
characteristic  is  undesirable. 


Figure  44:  Dependoicy  on  the  axes  chosen 


A2.  Mathematical  Extension  of  Median  to  Multiple  Dimensions 


The  problem: 

Given  a  set  of  vectors  Pi ,  P2, . . . ,  P..  Select  a  vector  out  of  them  which  is  the  median  vector. 

Definition  of  vector  median:  Let  denotes  the  distance  between  Pi  and  Py,  and  Si  =  HU  dij.  The  median 

vecttx^  is  the  Pm  whose  summation  Sm  is  the  smallest  among  Si  (1  <  <  <  n). 


A2.1.  Vector  Median  in  One  Dimension 

The  foregoing  definition  of  vector  median  is  exactly  the  same  as  the  scalar  median  in  one  dimension.  This  is 
depicted  in  Figure  45.  Three  points  Pi.  Pj,  and  Pa  are  at  the  same  position  on  the  left  P4  is  at  the  distance  a  from 
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Pi.  P2.  or  Pa.  Ps  is  far  away  on  the  right  at  the  disumce  b  firom  P4.  The  summations  of  the  distances  to  all  otha 
points  for  Pi,  P2,  P3,  P4.  and  Pj  are  2a  +  b,  2a  +  b,  2a+b,  3a  +  b,  and  3a  +  4b,  respectively.  Hence,  the  median 
vector  is  Pi,  P2,  or  P3.  This  is  the  same  as  the  scalar  median.  It  should  be  noticed  that  vector  median  does  not 
necessarily  select  the  pixel  that  is  the  closest  to  mean,  hi  the  Hgure  45,  Pi,  P2,  or  P3  is  not  the  closest  point  to 
the  mean.  The  mean  will  be  on  the  right  side  of  P4,  since  a<.b. 


Median  vector 
Pi  PaPj 


Mean 

I 


a«b 

Si  H  Se  ■  S3 ..  aa^fb 
S4>  3a.»b 
Ss-3a+4b 

Median  vector  is  P|.  P2>  or  P3. 
since  Si-  S2-  S3<  S4<  Sj 


Ps 

-• 


Figure  45:  Vector  median  in  one  dimensional  space 


Vector  Median  in  two  Dimensions 

A  two  dimensional  case  is  illustrated  in  Figure  46,  where  five  points  are  at  the  vertices  of  a  regular  pratagon  excqit 
Pi  which  is  slightly  inward.  The  length  of  the  side  is  a.  The  loigth  of  the  diagonal  line  is  b.  The  distances  from 
P]  to  the  closer  points  and  to  the  fartho’  points  are  o'  and  b',  respectively.  Intuitively,  Pi  should  be  the  median 
vector,  because  it  is  inward.  The  summations  of  the  distances  to  all  other  points  for  Pi,  P2,  Ps.  P4.  and  Ps  are 
2d'  +  2b^,  a  +  a'+2b,  2d  +  ft  +  b',2d  +  b  +  b',  and  a  +  o'  +  2b.  Since  o'  <  a  md  1/  <  b,  Si  is  the  smallest.  Thus, 
the  medina  vector  is  Pi  which  follows  the  intuitioa  It  is  intoesting  that  a  unique  median  vector  is  usually  found 
among  even  number  of  vectors  in  multiple  dimensions. 


Median  vector 

k 

P. 


P. 

a'  <  a  and  b'  <  b 
S|  a  2aV2b* 

a+aV2b 

S3  s  S4  v  2a-fb4b' 

Madian  vector  is  Pi, 
since  Si  <  Sjs  S4<  S3  . 

Figure  46:  Vector  median  in  two  dimensional  space 


A2  J.  Vector  Median  in  three  Dimensions 


The  example  in  Figure  43  is  examined  in  Figure  47.  The  summations  are  Smatnea  -  255(2v^  +  \/3)  »  1 163, 
Sey<m  =  SytUow  -  255(1  +  3\/2) «  1337,  and  5,w«,  =  255(2  +  2\/3) «  1393.  Therefore,  Pmaf«iiM  is  the  median  vector. 
The  median  vector  is  attracted  by  points  in  close  proximity. 
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Figure  47:  Vector  median  in  three  dimensional  space 


A3.  Color  Median  Filter 

Applying  the  vector  median  to  color  images  is  straightforward.  The  coltv  median  filter  merely  replaces  the  color 
pixel  at  a  point  by  the  median  vector  of  the  coltH*  pixels  in  a  neighborhood  of  the  point  In  the  color  case,  the 
number  of  pixels  in  the  filtering  window  is  not  necessarily  odd,  since,  in  two  or  more  dimensional  space,  a  median 
vector  is  uniquely  determined  among  the  even  number  of  vectors  that  is  greater  than  or  equal  to  four. 

By  definition,  the  color  median  filter  does  not  introdu^  new  color  pixels.  This  is  preferable  and  enhances  the 
edge  {Reserving  advantage  of  the  median  filter.  It  is  also  indei)endent  of  the  color  axes  chosen,  since  it  depends 
only  on  the  distances  among  color  pixels. 


A4.  Evaluation  of  Color  Median  Filter 

Both  the  color  median  filter  and  the  method  which  smoothes  each  com|)onent  of  the  image  separately  are  applied  on 
a  car  image  in  Figure  48.  The  window  size  is  3x  3.  Edge  {nt^Ies  of  the  red  comixxient  origi^,  the  red  component 
after  median  filtering,  in  the  row  240  are  in  Figure  49  and  figure  50,  resi)ective]y. 

At  the  column  220  in  Hgure  50,  a  sharp  intensity  valley  is  presmed  by  the  coltx’  median  filter,  while  it  is 
corrupted  by  the  other  method  which  smoothes  each  component  separately.  Mos'  of  the  {jeaks  are  ^tparently  acute 
in  the  results  of  the  color  median  filter.  But  at  the  column  410  in  Figure  50,  a  ..uctuation  in  the  middle  of  the  steep 
slope  is  smoothed  more  by  the  cdor  median  filter  than  the  other  method,  because  of  the  influences  by  the  green 
and  blue  comfxments.  This  shows  that  colw  median  filler  does  not  merely  {)ass  high  frequency  but  has  a  strong 
preference  when  picking  median  vector  {jixels. 


A5.  Conclusion 

The  cokv  median  filter  naturally  {Reserves  the  edges  of  images  more  than  the  method  which  smoothes  each 
component  of  the  image  separately  by  scalar  median  filler. 
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Figure  49:  Original  edge  profile  (Red  tow240) 


Top:  Median  filter  used  on  each  component  independently 
Bottom:  Color  median  filter 

Figure  SO:  Edge  profiles  after  median  filtering  (Red  iow240) 
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Car  Recognition  for  the  CMU  Naviab 

Karl  Kluge  and  Hideki  Kuga 


1.  Introduction 

LASSIE  (Landmark  yAcquisition  SlibSystem  Arproving  Exploration)  is  a  program  which  recognizes  a  car 
from  several  points  of  view.  Trapezoids  are  formed  from  groups  of  lines  extracted  from  a  color  image; 
instances  of  the  car  model  are  found  by  searching  for  sets  of  trapezoids  which  fit  the  constraints  between 
parts  of  the  car  such  as  the  windows,  roof,  and  trunk.  No  information  is  used  about  the  dimensions  of  the 
various  parts,  setting  LASSIE  apart  from  many  other  programs  which  use  features  extracted  from  images 
to  recognize  objects. 

The  remainder  of  the  introduction  sketches  out  the  varied  functions  of  object  recognition  in  the 
navigation  system  of  an  autonomous  vehicle  and  discusses  the  choice  of  domain  for  this  work.  The  next 
section  of  the  report  discusses  related  work,  followed  by  a  section  giving  a  detailed  description  of  the 
LASSIE  program.  Results  are  shown  for  a  number  of  images.  The  report  closes  with  a  discussion  of 
directions  for  further  work  on  LASSIE. 


1.1.  The  function  of  object  recognition  In  autonomous  navigation 

An  autonomous  vehicle  navigating  through  the  world  may  need  to  recognize  a  variety  of  natural  and 
man-made  objects  for  a  number  of  reasons: 

•  to  confirm  or  refine  its  internal  estimate  of  its  position  in  the  world  (Am  /  really  where  I  think  I 
am?) 

•  to  make  decisions  about  mission  execution  based  on  the  presence  or  absence  of  objects  in 
specified  locations  (/  should  halt  and  look  for  conflicting  traffic  if  there  is  an  octagonal  red 
sign  at  the  intersection  I'm  approaching.) 

•  to  gather  information  for  report  at  the  end  of  a  mission  (/  should  count  the  number  of  cars  I 
see  as  I  go  along  this  stretch  of  road.) 

•  to  update  the  navigation  system's  internal  model  of  the  world  (Remember  there  was  a  stop 
sign  at  that  intersection  so  I'll  know  that  I  should  plan  to  stop  and  look  for  traffic  the  next  time 
I  approach  it.) 

•  to  make  its  way  along  the  path  specified  by  its  mission.  Humans  tend  to  give  directions  to  an 
objective  in  high  level  qualitative  terms  (go  down  the  road  until  you  come  to  the  third  traffic 
light,  turn  right,  keep  going  until  you  see  the  a  big  church  on  the  left,  then  turn  left  at  the  next 
street...)  rather  than  giving  them  in  terms  of  precise  distances  (go  down  the  road  0.75  miles, 
turn  right,  go  another  2.3  miles,  turn  left...). 

A  navigation  system  may  select  an  object  to  serve  as  a  landmark  in  response  to  internal  goals.  As  an 
example,  the  navigation  system  might  realize  that  its  wheels  were  slipping  and  choose  a  stationary  object 
in  the  environment  in  order  to  keep  track  of  its  position  relative  to  the  object.  Alternatively,  the  mission 
may  specify  objects  for  which  the  system  is  to  watch.  Examples  of  this  would  be  Turn  right  at  the  third 
stop  sign,"  or  "when  you  get  to  the  big  yellow  marker  you  will  be  at  world  position  (x,  y,  z):  update  your 
internal  position  estimate  accordingly." 

Objects  to  be  recognized  may  be  artificial  (such  as  road  intersections,  buildings,  and  vehicles),  or  they 
may  be  natural  (such  as  distinctive  rock  formations  or  trees  or  rivers).  The  objects  may  have  semantic 
meaning  (this  is  a  car,  this  is  a  stop  sign,  this  is  a  tree),  or  they  may  simply  be  distinctive  structures 
detected  by  the  navigation  system’s  perceptual  modules  and  tracked  through  time  without  identifying 
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them  as  any  particular  kind  of  thing. 


1.2.  Choice  of  domain 

Object  recognition  may  be  called  upon  by  a  broad  range  of  capabilities  built  into  a  navigation  system, 
as  discussed  in  the  previous  section.  The  Sidewalk  Navigation  System  developed  for  the  Terregator  at 
CMU  [5]  used  segmentations  of  color  and  range  images  to  recognize  stairs  and  intersections  in  order  to 
navigate  around  a  network  of  campus  sidewalks.  In  order  to  test  other  types  of  capabilities  requiring 
object  recognition,  such  as  deciding  which  way  to  turn  at  an  intersection  based  on  the  presence  or 
absence  of  a  specified  object,  it  was  necessary  to  have  a  program  capable  of  finding  other  kinds  of 
objects.  The  planning  part  of  such  capabilities  can  be  tested  and  demonstrated  using  simple  objects 
(such  as  orange  traffic  cones).  A  module  capable  of  recognizing  a  more  challenging  type  of  object  is 
more  desirable,  both  as  a  much  more  challenging  recognition  problem  to  solve,  and  as  more  likely  to 
bring  out  unexpected  interactions  between  parts  of  the  navigation  system  as  the  recognition  module 
competes  for  system  resources. 

Cars  seemed  an  attractive  domain  for  a  number  of  reasons.  The  main  nontechnical  reasons  were  that 
cars  are  fairly  common  objects  which  are  easily  moved  and  weather  proof,  making  possible  the  easy 
collection  of  a  variety  of  images  to  use  as  data.  The  main  technical  reasons  were 

•  cars  differ  a  great  deal  in  size  and  appearance  (think  of  the  differences  between  the 
appearance  of  a  station  wagon  versus  a  four-door  sedan,  versus  a  compact  car),  but  the 
variation  of  appearance  is  much  less  than  for  a  domain  such  as  tree  recognition; 

•  there  is  a  great  deal  of  structure  in  the  edges  projected  into  an  image  from  a  car,  with  many 
of  the  edges  forming  quadrilaterals  corresponding  to  meaningful  parts  of  the  car; 

•  the  significant  linear  features  that  form  a  car  can  be  detected  reasonably  well  by  current  line 
extraction  techniques,  but  not  so  well  that  the  system  can  assume  perfect  or  close  to  perfect 
edge  data. 


1.3.  The  goals  and  state  of  the  research 

The  goal  of  this  research  is  to  produce  an  object  recognition  program  that 

•  is  capable  of  extracting  the  significant  geometric  structures  (in  this  case,  trapezoids  and  their 
subclasses  such  as  parallelograms  and  rectangles)  formed  by  the  edge  segments  extracted 
from  an  image; 

•  uses  the  relationships  between  those  stnjctures  and  constraints  from  an  object  model  to  find 
instances  of  the  object; 

•  does  not  use  more  than  the  crudest  of  information  about  the  actual  dimensions  of  the  objects 
(for  instance  discriminating  station  wagons  from  compacts  based  on  the  length  of  the  body 
relative  to  the  length  of  the  hood,  or  stating  that  the  trunk  of  a  car  projects  into  the  image  as  a 
parallelogram,  which  imposes  some  crude  constraints  on  the  dimensions  of  sides  of  the 
tnjnk). 

This  approach  is  in  contrast  to  many  programs  that  perform  object  recognition  based  on  a  precise  or 
parameterized  model  of  the  shape  of  the  object(s)  to  be  recognized.  With  an  exact  shape  model,  or  even 
a  parameterized  model,  a  previously  unseen  car  may  not  be  recognizable.  The  hope  is  that  using  only  the 
crudest  kind  of  metrical  information  about  the  objects  will  allow  a  more  generic  recognition  capability. 

A  variety  of  images  of  a  particular  car  were  used  in  testing  the  program  as  it  evolved  to  its  current  state. 
The  program  demonstrates  a  fair  amount  of  robustness  in  processing  those  images,  as  will  be  seen  from 
the  results  in  section  4.  Recognition  of  different  makes  of  cars  has  not  been  attempted,  and  will 
undoubtedly  require  a  fair  amount  of  modification  of  the  model  of  the  parts  of  the  car,  but  should  be 
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possible  within  the  framework  that  has  been  developed. 


2.  Related  Work 

In  recent  years  much  of  the  research  in  object  recognition  has  been  concerned  with  the  recognition  of 
objects  based  on  data  from  3-0  sensors  such  as  laser  range  finders.  For  an  extensive  survey  of  such 
work  see  BesI  and  Jain  [1], 

Brooks  [2]  describes  ACRONYM,  a  system  which  recognized  objects  (represented  as  collections  of 
parameterized  generalized  cones)  using  edges  extracted  from  an  intensity  image.  Edges  were  collected 
into  "ribbons"  which  represented  contours  of  generalized  cones.  A  constraint  management  system  (CMS) 
was  used  to  verify  the  consistency  of  hypothesized  matches  between  "ribbons"  extracted  from  the  image 
and  object  parts.  A  match  was  consistent  if  there  was  some  viewpoint  from  which  the  match  could  have 
been  imaged,  with  constraints  on  possible  camera  position  accumulated  based  on  the  parameters  of  the 
generalized  cones  matched  and  the  parameters  of  the  corresponding  image  ribbons. 

Goad  [4]  created  a  system  which  precompiled  an  exact  description  of  the  lines  forming  an  object  into  a 
recognition  program  for  that  object.  The  programs  would  search  the  lines  extracted  from  an  image  for 
matches.  The  object  did  not  have  to  be  a  polyhedron,  it  simply  had  to  have  enough  linear  features  for 
matching  to  take  place.  Matching  was  done  via  a  simple  backtracking  search  which  checked  sets  of  lines 
for  matches  based  on  their  being  consistent  with  some  viewpoint.  Rather  than  use  a  CMS  as  ACRONYM 
did,  the  program  made  assumptions  which  allowed  it  to  map  possible  viewpoints  onto  a  tessellation  of  the 
unit  sphere.  The  set  of  cells  corresponding  to  possible  viewing  directions  would  be  restricted  to  reflect  the 
new  constraints  on  camera  position  as  lines  were  added  to  a  match. 

SCERPO  (Spatial  Correspondence,  Evidential  Reasoning,  and  Perceptual  Organization),  described  in 
Lowe  [7],  used  estimates  of  the  likelihood  of  accidental  alignments  to  look  for  pairs  of  parailel  lines  and 
pairs  of  lines  whose  endpoints  were  close  enough  to  suggest  their  being  connected.  It  then  looked  for 
pairs  of  parallel  lines  connected  to  other  pairs  of  lines  to  form  trapezoids.  These  trapezoids  were  matched 
to  parts  of  the  object  being  looked  for  to  give  an  initial  pose  estimate.  The  program  then  searched  for 
additional  lines  to  confirm/refute  the  initial  match  and  refine  the  pose  estimate.  Like  Goad's  system,  it 
required  an  exact  model  of  the  geometry  of  the  lines  forming  the  object. 

Thompson  and  Mundy[9]  describe  a  scheme  for  recognizing  objects  using  a  primitive  called  the 
"vertex  pair."  Approximating  the  projection  from  model  to  image  by  an  affine  transform,  a  match  between 
a  vertex  pair  in  the  model  and  its  projection  in  the  images  completely  specifies  the  pose  of  the  object. 
Matches  are  found  by  a  fairly  brute-force  voting  scheme  in  the  six  dimensional  parameter  space.  Once 
again,  an  exact  description  of  the  object's  geometry  is  required. 

Ohta's  system  [8]  used  color,  texture,  shape,  and  adjacency  properties  of  regions  extracted  from  a 
color  image  to  label  the  regions  as  parts  of  various  kinds  of  objects  such  as  road,  car,  tree,  or  building. 
The  program  had  no  concept  of  the  structure  of  the  objects  in  a  scene  —  a  set  of  adjacent  regions 
labeled  "building"  might  be  parts  of  one  building  or  of  several  buildings. 

The  3-D  MOSAIC  system  described  by  Herman  and  Kanade  [6]  worked  in  the  domain  of  aerial  images 
of  buildings.  It  extracted  line  segments  from  the  scene  and  grouped  them  together  into  sets  of  segments 
forming  faces  of  buildings  in  the  scene.  Monocular  and  binocular  cues  were  combined  with  assumptions 
about  the  domain  (for  instance,  that  walls  are  usually  vertical  and  roofs  are  usually  parallel  to  the  ground 
plane)  to  constmct  a  wireframe  representation  of  the  scene.  The  system  had  no  recognition  component, 
but  the  ideas  used  in  grouping  segments  into  potential  building  faces  had  an  influence  on  the  grouping 
processes  in  LASSIE. 

The  system  described  by  Fua  and  Hanson  [3]  works  in  the  domain  of  aerial  image  analysis.  The  image 
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is  segmented  into  regions,  and  edges  are  extracted  using  the  Sobel  edge  operator.  Expectations  about 
the  geometry  of  "cultural  regions,"  mainly  building  outlines,  are  used  to  drive  the  resegmentation  of  the 
image  in  order  to  extract  cultural  regions.  These  expectations  are  very  weak  and  generic.  The  system  is 
capable  of  deducing  certain  things  about  the  relations  between  the  regions  it  finds,  for  instance,  that  a  lit 
region  joined  along  an  edge  to  a  darker,  non-shadow  region  is  likely  to  be  formed  by  a  peaked  roof. 


3.  The  LASSIE  object  recognition  program 


3.1 .  Overview 

The  recognition  process  consists  of  the  following  stages: 

•  Extract  edge  points  from  a  color  image,  then  link  them  into  line  segments. 

•  Merge  segments  that  are  almost  collinear  and  are  separated  by  a  small  gap. 

•  Eliminate  short  line  segments. 

•  Extract  significant  line  pairs.  The  pairs  are  chosen  based  on  their  either  being  nearly  parallel 
or  possibly  forming  the  two  non-parallel  sides  of  a  trapezoid. 

•  Examine  the  pairs  formed  in  the  previous  step  to  look  for  segments  that  would  close  the  pairs 
off  to  form  trapezoids. 

•  Examine  the  traprezoids  found  in  the  previous  step  to  merge  sets  of  overlapping  trapezoids 
whose  shapes  are  similar. 

•  Search  for  groups  of  trapezoids  potentially  formed  by  fhe  windows,  roof,  and  trunk  of  a  car. 

•  Search,  given  an  initial  match,  for  other  features  (ellipses  corresponding  to  wheels,  lines  from 
the  back  and  side  of  the  car)  to  confirm  that  match. 

Apart  from  the  final  verification  stage,  the  system  is  currently  purely  a  bottom-up  program,  with  no  top- 
down  interactions  between  the  earlier  stages. 


3.2.  Description  of  the  segmentation  and  grouping  stages 

Segmentation 

•  A  blue  minus  red  (B  -  R)  image  is  computed  from  the  input  color  image. 

•  The  Canny  edge  detector  is  run  over  the  B  -  R  image.  The  sigma  chosen  is  2.5. 

•  The  5%  of  image  points  with  highest  gradient  magnitude  are  selected  from  the  Canny  output 
as  edge  points. 

•  A  line  tracker,  written  at  CMU  by  Hiro  Miwa,  is  mn  over  the  thresholded  edge  image.  The  line 
tracker  places  breakpoints  in  chains  of  edge  pixels  by  looking  for  places  where  the  edge 
cun/ature  is  changing  rapidly  at  multiple  scales.  Gaps  of  one  pixel  in  edges  are  jumped  by 
the  tracking  program.  Lines  shorter  than  10  pixels  are  discarded.  The  line  tracker  also  finds 
ellipses  in  the  edge  data  which  can  be  used  to  find  the  location  of  wheels  forming  part  of  a 
car. 


Line  merging 

Segmentation  may  break  up  lines  in  the  scene  because  of  gaps  in  the  thresholded  edge  image.  Also, 
very  close  parallel  edge  segments  may  arise  from  the  same  linear  feature  in  the  scene.  A  simple  merging 
procedure  is  applied  to  compensate  for  these  errors  in  edge  extraction,  illustrated  in  figure  3-1 .  Given  two 
edge  segments.  Si  and  S2: 

•  A  line  is  fit  to  Si  and  S2  using  least  squares.  The  endpoints  of  Si  and  S?  are  projected  onto 
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that  line  in  order  to  find  the  endpoints  of  the  merged  segment,  S3. 

•  SI  and  S2  are  merged  to  form  S3  if  the  following  conditions  are  met; 

1 .  the  distance  between  the  endpoints  of  Si  and  S2  and  the  merged  segment  S3  is  less 
than  some  number  of  pixels,  and 

2.  SI  and  S2,  when  projected  onto  S3,  either  overlap  or  are  separated  by  a  gap  smaller 
than  a  threshold  set  by  the  user. 


^Une 


Figure  3-l :  Line  merging 
Thresholding 

The  lines  are  thresholded  on  length  to  eliminate  spurious  lines  created  by  such  things  as  variations  in 
the  surface  of  the  road  on  which  the  car  is  resting,  and  specular  reflections  of  tree  foliage  off  the  surface 
of  the  car.  The  default  threshold  is  30  pixels. 

Examination  of  line  pairs 

The  set  of  lines  is  examined  for  pairs  of  lines  which  are  parallel  within  some  tolerance,  or  which  could 
be  the  non-parallel  sides  of  a  trapezoid,  i.e.,  the  virtual  lines  connecting  corresponding  endpoints  of  the 
segments  are  close  to  parallel. 

Extraction  of  trapezoids 

Each  pair  of  lines  found  in  the  last  stage  could  possibly  be  two  of  the  four  sides  of  a  trapezoid.  To  look 
for  evidence  supporting  one  of  the  sides  connecting  the  pair  to  form  a  trapezoid 

•  the  list  of  edge  segments  is  searched  for  edges  which  meet  the  following  criteria:  they  are 
close  to  parallel  to  the  hypothesized  side;  their  endpoints  are  closer  than  a  threshold  (based 
on  the  separation  between  the  line  pair)  to  the  hypothesized  side;  and  whose  probability  of 
being  a  false  match  to  the  hypothesized  side  is  less  than  some  threshold  (using  the  formula 
given  by  Lowe  [7]); 

•  the  supporting  segments  found  are  projected  onto  the  hypothesized  side  and  the  fraction  of 
the  hypothesized  side  covered  is  computed 

•  a  line  is  fit  to  the  supporting  segments  if  a  sufficient  fraction  of  the  hypothesized  side  is 
covered  (see  figure  3-2).  If  only  one  of  the  two  missing  sides  is  found  by  this  process  the 
other  is  hypothesized  as  present. 

A  number  of  heuristics  are  used  to  eliminate  some  of  the  trapezoids  found  by  this  process.  A  trapezoid 
is  discarded  if  any  of  the  following  conditions  are  met; 

•  one  side  of  the  trapezoid  does  not  have  sufficient  support  and  is  hypothesized,  and  an  edge 
segment  crosses  the  hypothesized  side  and  extends  more  than  a  threshold  distance  outside 
the  trapezoid; 

•  the  shorter  side  of  a  pair  of  opposite  sides  is  less  than  some  fraction  of  the  length  of  the 
longer  side  of  the  pair; 

•  the  angle  between  a  pair  of  adjacent  sides  is  greater  than  a  threshold  angle; 

•  a  side  supported  by  one  of  the  original  pair  of  lines  is  longer  than  some  multiple  of  the  length 
of  the  original  segment. 
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These  heuristics  are  illustrated  in  figure  3-3. 


distance  a  extends  outside  >  thr 


(a/b)<tlir 


angle  a>tlir  (b/a)>thr 

Figure  3-3:  T rapezoid  elimination  heuristics 
Trapezoid  merging 

The  line  pair  finding  process  wiil  find  several  pairs  of  lines  from  the  set  of  edges  fomning  a  trapezoid  in 
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the  image.  In  the  simplest  case,  the  top  and  bottom  edges  will  form  one  pair,  and  the  left  and  right  edges 
will  form  another  pair.  This  will  result  in  the  finding  of  two  trapezoids,  each  using  the  same  set  of  lines. 
These  are  really  the  same  trapezoid.  The  duplication  should  be  detected,  and  the  two  trapezoids  merged. 
In  addition,  things  such  as  trim  around  windows  may  create  multiple  pairs  of  lines  which  then  give  rise  to 
trapezoids  that  are  very  similar.  This  stage  of  processing  looks  through  the  list  of  trapezoids  returned  by 
the  previous  stage  and  merges  pairs  of  trapezoids  which  have  corresponding  sides  which  are  close  to 
each  other,  nearly  parallel,  and  less  that  some  probability  of  being  false  matches  to  each  other  (once 
again  using  Lowe’s  formula  [7]). 


3.3.  Feature-fetchers  and  the  search  for  Initial  matches 

The  procedures  used  to  search  for  initial  matches  of  image  features  to  instances  of  cars  consist  of  a 
match  conflict  resolver  and  a  set  of  procedures  called  feature-fetchers  (see  figure  3-4).  The  match 
conflict  resolver  keeps  track  of  "claims"  to  image  features  by  matches.  Oniy  one  match  using  a  given 
image  feature  (the  match  with  the  highest  confidence)  survives,  with  the  confidence  values  of  competing 
matches  for  an  image  feature  getting  set  to  zero.  Feature-fetchers  are  procedures  which  return  instances 
of  particular  features.  The  features  returned  may  be  primitive  features  (such  as  iines,  ellipses,  or 
trapezoids)  or  compound  features  (such  as  a  pair  of  trapezoids  which  could  match  the  side  and  rear 
windows  of  a  car).  The  model  of  the  appearance  of  a  car  is  encoded  procedurally  in  the  feature-fetchers 
in  the  form  of  calls  to  functions  that  evaluate  constraints  that  should  hold  for  a  match. 

A  feature-fetcher  may  be  called  upon  multiple  times  to  return  a  given  feature  instance.  A  good  example 
of  this  is  the  trapezoidjist  feature-fetcher,  which  is  called  by  all  the  other  feature-fetchers.  Each  instance 
of  a  feature  found  by  a  feature-fetcher  is  assigned  a  sequential  index  number.  That  allows  each  feature- 
fetcher  calling  trapezoidjist  to  keep  track  of  which  trapezoids  it  has  seen  by  storing  the  index  of  the  last 
instance  returned  to  it  by  trapezoidjist  without  requiring  trapezoidjist  to  know  anything  about  what 
feature-fetchers  are  cailing  it.  in  the  case  of  the  trapezoidjist  feature-fetcher  ail  the  instances  of  the 
feature  are  extracted  before  the  matching  process  begins.  In  the  case  of  the  other  feature-fetchers 
instances  are  found  as  needed  and  cached.  An  instance  can  be  fetched  from  the  previously  cached 
results  if  another  feature-fetcher  calls  requesting  an  instance  with  an  index  number  beiow  the  highest 
index  value  assigned  so  far.  This  avoids  duplication  of  work  by  the  feature-fetchers. 

The  functions  of  the  individual  feature-fetcher  procedures  are  as  follows: 

•  trapezoidjist  returns  trapezoids  extracted  in  the  earlier  stages  of  the  program 

•  get_windows  returns  a  partial  car  match  containing  a  pair  of  trapezoids  which  could 
correspond  to  the  side  and  rear  windows  of  a  car  instance 

•  add_roof  gets  partial  matches  from  get_windows  and  looks  for  trapezoids  from  trapezoidjist 
which  could  match  the  roof  given  the  trapezoids  in  the  match  of  the  windows 

•  addjrunk  gets  partial  matches  from  add_roof  and  looks  for  trapezoids  from  trapezoidjist 
which  could  match  the  trunk  given  the  window  and  roof  trapezoids. 

Feature-fetchers  may  employ  multiple  strategies  to  do  their  jobs.  The  addjrunk  feature-fetcher  is  a 
good  example.  The  roof  and  windows  of  the  car  look  very  similar  from  either  the  left  or  right,  appearing 
roughly  like  a  truncated  pyramid.  Since  add_roof  can't  disambiguate  which  side  of  the  car  is  being  looked 
at,  addjrunk  employs  two  strategies  to  look  for  a  trunk  trapezoid  to  extend  a  partial  match  from  add_roof. 
In  the  first  strategy  it  assumes  the  car  is  being  seen  from  the  right  and  looks  for  the  trunk  to  the  left  of  the 
windows.  When  no  more  instances  are  found  by  the  first  strategy,  the  second  strategy  is  invoked.  This 
strategy  assumes  that  the  car  is  being  seen  from  the  left  side,  and  looks  for  the  trunk  to  the  right  of  the 
windows.  This  involves  stepping  through  the  list  of  instances  found  by  add_roof  a  second  time,  taking 
advantage  of  the  caching  done  by  the  feature-fetchers. 
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Figure  3-4:  Hierarchy  of  search  procedures 


The  feature-fetchers  use  a  variety  of  constraints  on  the  relationships  between  trapezoids  and  between 
sides  of  trapezoids  to  find  matches.  The  routines  which  check  if  the  constraints  are  satisfied  take  into 
account  uncertainty  about  the  objects  (for  instance,  whether  both  endpoints  of  a  segment  are  known  for 
sure).  Currently  eight  constraints  are  used  in  the  model: 

•  rightp(seg1,  seg2)  --  Weak  "right-or  constraint  between  segments.  Satisfied  if  the 
maximum  image  column  number  of  either  endpoint  of  segl  is  greater  than  the  maximum 
image  column  number  of  either  endpoint  of  seg2.  Othenwise  not  satisfied.  This  definition 
assumes  that  the  camera  is  roughly  vertical,  and  this  constraint  is  used  very  sparingly  to 
minimize  the  restrictions  this  places  on  the  camera. 
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•  leftp(segl ,  seg2)  --  Weak  "left-of"  constraint  between  segments.  Satisfied  if  the  minimum 
image  column  number  of  either  endpoint  of  segl  is  less  than  the  minimum  image  column 
number  of  either  endpoint  of  seg2.  Otherwise  not  satisfied.  Also  used  sparingly  for  the  same 
reason  as  given  for  rightp. 

•  areparallel(segl ,  seg2,  tol)  --  Parallelism  constraint  between  segments.  Satisfied  if  seg1 
and  seg2  are  parallel  within  the  tolerance  given  by  tol.  May  be  satisfied  if  the  segments  aren't 
close  to  parallel,  but  both  endpoints  of  either  one  of  the  segments  are  hypothesized. 
Otherwise  it  is  not  satisfied. 

•  angbetween(seg1,  seg2,  low,  high)  --  Included  angle  constraint  between  segments.  Here 
seg1  and  seg2  are  directed.  Satisfied  if  the  angle  going  counterclockwise  from  seg1  to  seg2 
is  between  the  values  low  and  high.  May  be  satisfied  if  the  actual  direction  of  either  segment 
is  not  known  because  both  its  endpoints  are  hypothesized.  Otherwise  not  satisfied. 

•  trapform(trapezoid,  start_side,  angjypel,  ang_type2,  angjypeS,  ang_type4)  -  Rough 
shape  constraint  for  trapezoids.  The  ang_type  variables  have  the  value  of  OB  (for  obtuse  -- 
defined  here  as  an  angle  between  70  and  180  degrees)  or  AC  (for  acute  -  defined  here  as 
an  angle  between  0  and  110  degrees).  Trapform  checks  the  angle  at  each  vertex  (with 
start_side  used  to  indicate  what  side  to  consider  side  zero)  to  see  if  it  is  of  the  appropriate 
type.  Trapform  is  satisfied  if  the  angles  at  all  four  vertices  are  of  the  correct  type.  It  may  be 
satisfied  if  the  angle  at  a  vertex  is  of  the  wrong  type  but  the  position  of  the  vertex  is 
hypothesized.  Othenwise  it  is  not  satisfied. 

•  phratlo(trap1 ,  pain ,  htypel ,  trap2,  pair2,  htype2,  low,  high)  --  Relative  height  ratio  constraint 
between  trapezoids.  Trapt  and  trap2  are  trapezoids,  pair1  and  pair2  specify  which  of  the  two 
pairs  of  opposing  sides  is  involved  from  trapi  and  trap2.  Htypel  and  htype2  specify  the 
segment  pair  distance  measure:  either  midpoint-to-midpoint  for  nonparallel  lines  or  midpoint- 
to-opposite-segment  for  parallel  or  roughly  parallel  segments.  The  ratio  between  the  height 
of  the  pair  specified  for  trapi  and  the  height  of  the  pair  specified  for  trap2  is  computed.  If  the 
ratio  falls  between  the  low  and  high  values  specified  the  constraint  is  satisfied.  If  the  ratio 
falls  outside  the  specified  range,  but  one  of  the  segments  of  one  of  the  specified  pairs  was 
only  hypothesized,  then  the  constraint  may  be  satisfied.  Otherwise  the  constraint  is  not 
satisfied. 

•  paraface(seg1 ,  low1 ,  high1 ,  seg2,  low2,  high2)  --  Constraint  on  the  overlap  between  parallel 
or  roughly  parallel  segments.  The  endpoints  of  seg2  are  projected  onto  the  line  specified  by 
seg1  and  the  fraction  of  the  length  of  segl  overlapped  by  the  projection  of  seg2  is  computed. 
Similarly  the  fraction  of  seg2  overlapped  by  the  projection  of  seg1  is  computed.  The 
constraint  is  satisfied  if  the  fraction  of  segl  overlapped  is  between  lowl  and  highi  and  the 
fraction  of  seg2  overlapped  is  between  low2  and  high2.  The  constraint  may  be  satisfied  if  the 
overlapped  fraction  for  a  segment  is  out  of  range  but  the  length  of  the  segment  is  not  know 
for  sure  (because  one  or  both  of  its  endpoints  are  hypothesized).  Otherwise  the  constraint  is 
not  satisfied. 

•  onslde(frapl ,  edge,  trap2)  --  Constraint  on  the  relative  position  of  two  trapezoids.  Picture 
walking  clockwise  around  the  sides  of  trapi .  The  constraint  is  satisfied  if  the  side  of  trapi 
specified  by  edge  is  the  only  side  such  that  all  four  vertices  of  trap2  are  to  your  left  as  you 
walk  along  the  side.  Otherwise  the  constraint  is  not  satisfied. 

The  above  routines  return  a  confidence  factor  between  0.0  and  1 .0.  Currently  they  return  1 .0  if  the 
constraint  is  satisfied;  0.1  if  the  constraint  may  be  satisfied  as  described  in  the  above  descriptions;  and 
0.0  if  the  constraint  was  not  satisfied.  Any  constraint  on  a  possible  match  evaluating  to  0.0  eliminates  that 
match.  The  overall  confidence  of  a  match  is  just  the  sum  of  the  values  returned  by  the  calls  checking 
constraints  from  the  model  divided  by  the  number  of  constraints. 
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3.4.  Verification  of  Initial  matches 

The  verification  stage  goes  through  the  list  of  matches  which  survive  match  conflict  resolution,  and 
checks  for  lines  and  ellipses  verifying  features  that  were  not  detected  bottom-up.  Information  about 
expected  orientation  and  position  of  lines  based  on  a  partial  match  is  used  to  constnjct  search  windows  to 
detect  lines  belonging  to  the  rear  and  side  of  the  car.  The  commitment  to  making  no  assumptions  about 
the  sizes  of  various  parts  is  broken  in  the  verification  stage,  which  assumes  that  the  side  of  the  car  is 
within  some  range  of  ratios  of  the  height  of  the  side  window.  This  violation  of  one  of  the  design  goals  can 
probably  be  eliminated  by  looking  for  the  wheels  and  the  bottom  of  the  side  of  the  car  together.  Ellipses 
extracted  from  the  edge  data  are  examined  to  look  for  wheels.  The  results  of  this  stage  are  the  final 
matches  found  by  the  program. 


4.  Results 

Figure  4-1  shows  the  lines  extracted  from  an  image  of  the  car  in  the  upper  left  section;  the  lines  after 
thresholding  and  merging  in  the  upper  right  section;  the  extracted  trapezoids  in  the  middle  left  section;  the 
initial  match  found  bottom-up  in  the  lower  left  section  ;  and  the  confirmed  match  in  the  lower  right  section. 
It  can  be  seen  that  the  line  merging  was  able  to  improve  the  quality  of  the  line  segments,  for  instance  by 
linking  together  the  lines  forming  the  back  edge  of  the  trunk  and  the  line  forming  the  top  edge  of  the  side 
window. 

The  trapezoids  found  include  the  roof,  the  taink,  the  rear  window,  and  bits  of  trim  along  the  side.  There 
are  also  a  number  of  trapezoids  found  which  arise  from  accidental  alignments  in  the  image.  The 
hypothesized  fourth  sides  on  a  number  of  them  pass  through  the  much  stronger  trapezoids  found  for  the 
tmnk  and  rear  window,  but  do  not  get  eliminated  because  no  edge  pierces  far  enough  through  the  virtual 
sides. 

Only  one  match  remains  after  match  conflict  resolution.  Comparing  the  final  match  with  the  original 
lines,  one  can  see  that  the  program  has  done  a  good  job  of  finding  the  various  parts  of  the  car. 

Figures  4-2  and  4-3  show  the  result  of  running  the  program  on  lines  extracted  from  an  image  of  the  car 
as  seen  from  the  other  side.  Here  two  matches  sun/ive  the  match  conflict  resolution,  even  though  there  is 
only  one  car  in  the  scene.  This  is  a  result  of  the  match  conflict  resolution  using  features  rather  than  image 
areas  as  the  primitive  things  claimed  by  a  match. 

Figures  4-4  and  4-5  show  the  results  produced  by  LASSIE  for  six  additional  images.  The  original  lines 
extracted  from  the  images  are  on  the  left  and  the  verified  matches  are  on  the  right.  As  can  be  seen,  the 
program  does  a  fairly  good  job  of  locating  the  parts  of  the  car  despite  noise  in  the  extracted  edges. 


5.  Directions  for  future  work 

Analysis  of  trapezoid  overlap  to  improve  match  conflict  resolution  and  heuristic  discarding  of 
trapezoids.  The  current  match  conflict  resolution  scheme  jnly  looks  at  features.  The  trapezoid  merging 
may  not  be  able  to  merge  all  of  the  trapezoids  arising  from  the  various  parts  of  the  car.  If  all  of  the 
features  searched  for  give  rise  to  two  or  more  trapezoids  then  multiple  matches  will  sun/ive  match  conflict 
resolution,  despite  there  only  being  one  car  in  the  image.  An  example  of  this  was  shown  in  the  results 
section  in  figure  4-3.  Analyzing  the  area  of  overlap  of  the  features  forming  the  matches  should  allow 
resolution  of  this  sort  of  conflict.  The  method  will  have  to  be  designed  in  a  way  that  will  handle  partial 
occlusion  of  one  car  by  another. 

Also,  in  some  cases  a  hypothesized  side  which  closes  off  a  trapezoid  goes  through  another  trapezoid 
for  which  there  is  very  strong  support.  There  are  cases  where  the  test  for  lines  crossing  hypothesized 
sides  fails  to  catch  this.  Looking  at  areas  of  intersection  between  trapezoids  would  help  in  the  detection 
of  these  cases. 


Figure  4-2:  Second  image:  original  lines,  merged  and  thresholded  lines,  and  extracted  trapezoids 

Improving  the  confidence  factor  scheme.  The  matches  which  survive  conflict  resolution  are  not 
always  the  most  aesthetically  pleasing  ones.  This  is  largely  due  to  the  overly  simple  confidence  measure 
scheme  currently  used. 

There  are  two  major  areas  in  which  the  confidence  factor  scheme  could  stand  improvement.  The  first  is 
in  the  granularity  of  the  values  returned  by  the  constraint  tests.  Currently  one  of  three  values  is  returned 
based  on  whether  the  constraint  is  satisfied,  unsatisfied  but  unsure,  or  definitely  unsatisfied.  This  is  too 
coarse.  For  instance  the  parallelism  constraint  could  return  a  value  proportional  to  how  close  to  parallel 
the  segments  were. 

The  second  area  for  improvement  is  the  evidence  combination  scheme,  which  is  completely  ad  hoc. 
Taking  the  product  of  the  values  /eturned  for  the  various  constraints  in  the  model  quickly  produces  values 
which  are  all  uniformly  small.  Other  techniques  have  not  been  investigated  at  this  point. 


Figure  4-3:  Second  image;  bottom-up  match  and  match  with  verification 

Integration  of  top-down  search  and  verification  Into  the  feature-fetchers.  Currently  top-down 
verification  of  features  not  found  in  partial  matches  is  done  in  a  separate  phase.  Feature-fetchers  should 
be  written  to  try  to  extend  a  partial  match  by  top-down  search  for  missing  features  if  no  matching  feature 
was  found  by  the  bottom-up  feature  extraction. 

Making  the  search  more  flexible.  Currently  there  is  a  single  fixed  order  in  which  features  are 
searched  for;  first  find  candidates  for  the  windows,  then  extend  by  adding  a  roof,  then  extend  by  adding  a 
trunk.  Even  with  top-down  search  capabilities  built  into  the  feature-fetchers  this  is  probably  not  sufficient. 
The  whole  idea  of  having  feature-fetchers  which  cache  their  results  was  to  allow  more  complex  search 
strategies. 

Doing  something  about  all  the  thresholds.  There  are  currently  a  dozen  thresholds  visible  to  the 
user,  with  another  three  or  four  buried  in  the  code.  Their  default  values  are  the  result  of  good  initial 
guesses  followed  by  experimenting  with  how  far  they  could  be  changed  before  the  results  started  to 
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deteriorate.  Setting  them  loosely  enough  to  find  all  the  features  needed  in  the  bottom-up  stage  of  the 
search  results  in  various  bogus  features  being  found  *&s  well.  Doing  an  initial  pass  with  very  tight 
thresholds  to  find  the  strongest  features  and  letting  the  individual  feature-fetchers  loosen  them  as  needed 
during  top-down  search  would  hopefully  make  the  system  less  sensitive  to  these  parameters. 

Other  views,  other  cars.  Currently  the  system  is  only  programmed  to  recognize  the  car  as  seen  from 
two  views,  from  behind  to  the  right  and  from  behind  to  the  left.  Extension  to  the  other  views  of  the  car 
needs  to  be  done.  Also  the  program  needs  to  be  extended  to  recognize  other  makes  of  cars. 


6.  Summary 

The  LASSIE  program  has  demonstrated  the  ability  to  robustly  extract  significant  groups  of  lines  forming 
trapezoids  in  the  test  images  we  have  used.  It  can  find  instances  of  the  test  car  from  two  views  using 
only  the  geometric  constraints  described,  without  any  model  of  the  dimensions  of  the  car  other  than  the 
one  exception  mentioned  in  the  section  on  match  verification  (section  3.4).  It  can  do  this  despite  the 
imperfections  of  edge  data  extracted  from  real  outdoor  images  of  the  test  car.  Extensions  of  the  work  to 
support  more  complex  search  strategies,  mixed  top-down  and  bottom-up  searching  in  the  feature- 
fetchers,  and  improved  confidence  measures  can  be  expected  to  improve  the  performance  of  the 
program. 
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Abstract 

Geometric  camera  calibration  is  tlie  process  of  determining  a  mapping  between  points  in  world 
coordinates  and  the  corresponding  image  locations  of  the  points.  In  previous  methods,  calibration  typically 
involved  the  iterative  solution  to  a  system  of  non-linear  equations.  We  present  a  method  for  poforming 
camera  calibration  that  provides  a  complete,  accurate  solution,  using  only  linear  systems  of  equations.  By 
using  two  calibration  planes,  a  line-of-sight  vector  is  defined  for  each  pixel  in  the  image.  The  effective 
focal  point  of  a  camera  can  be  obtained  by  solving  the  system  that  defines  the  intersection  point  of 
the  line-of-sight  vectors.  Once  the  focal  point  has  been  determined,  a  complete  camera  model  can  be 
obtained  with  a  straightforward  least  squares  procedure.  This  method  of  geometric  camera  calibration 
has  the  advantages  of  being  accurate,  efficient,  and  practical  for  a  wide  variety  of  applications. 
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1  Introduction 

Many  problems  in  computer  vision  and  graphics  require  mapping  points  in  space  to  corresponding  points  in 
an  image.  In  computer  graphics,  for  example,  an  object  model  is  defined  with  respect  to  a  world  coordinate 
system.  To  generate  an  image,  the  points  that  lie  on  the  visible  surfaces  of  the  object  must  be  mapped 
onto  the  image  plane;  that  is,  3d  world  points  must  be  mapped  onto  2d  image  points.  In  computer  vision, 
tte  image  locations  of  points  on  an  object  can  be  used  to  info*  three-dimensional  properties  of  the  object; 
in  this  case,  2d  image  points  must  be  mapped  back  onto  the  original  3d  world  points.  In  both  cases, 
the  mapping  between  3d  world  coordinates  and  2d  image  coordinates  must  be  known.  Geometric  camera 
calibration  is  the  process  of  determining  the  2d-3d  mapping  between  a  camera  and  a  world  coordinate 
system. 

We  decompose  the  general  problem  of  geometric  camera  calibration  into  two  subproblems; 

•  The  projection  problem:  given  the  location  of  a  point  in  space,  predict  its  location  in  the  image;  that 
is,  project  the  point  into  the  image. 

•  The  back-projection  problem:  given  a  pixel  in  the  image,  compute  the  line-of-sight  vector  through 
the  pixel;  that  is,  back-project  the  pixel  into  the  world. 

A  complete  solution  to  the  camera  calibration  problem  entails  deriving  a  model  for  the  camera  geometry 
that  permits  the  solution  of  both  the  projection  and  the  back-projection  problems.  For  many  applications, 
a  complete  solution  is  necessary.  Some  examples  from  the  domain  of  mobile  robots  will  help  illustrate  the 
problems. 

In  the  CMU  Navlab  project  [6],  a  robot  vehicle  follows  roads  using  data  from  a  color  TV  camera. 
In  each  image,  the  road  is  extracted,  and  the  centerline  and  direction  of  the  road  computed  in  image 
coordinates.  These  parameters  are  then  back-projected  into  vehicle  coordinates  and  used  to  plot  a  course 
for  the  vehicle  that  stays  within  the  road  boundaries. 

Turk,  et  al  [8]  describe  a  similar  road  following  technique  for  the  Autonomous  Land  Vehicle  (ALV). 
Rather  than  parameterizing  the  road  in  terms  of  cent^line  and  direction,  they  describe  the  road  boundaries 
as  a  sequence  of  points.  The  line-of-sight  vectors  for  each  of  the  boundary  points  are  computed  by  back- 
projection.  The  intersections  of  the  line-of-sight  vectors  with  the  ground  plane  yield  the  points  in  the  world 
between  which  the  robot  must  steer  to  stay  on  the  road.  In  addition,  the  ALV  needs  to  know  the  predicted 
position  of  the  road  in  each  image.  This  prediction  is  obtained  by  projecting  the  location  of  the  road  into 
each  image,  based  on  the  position  of  the  road  in  the  previous  image,  and  the  motion  of  the  vehicle  between 
images. 

The  simplest  model  fcv  camera  geometry  is  the  pinhole,  or  perspective  model.  See  figure  1.  Light  rays 
from  in  front  of  the  camera  converge  at  the  pinhole  and  are  projected  onto  the  image  plane  at  the  back  of 
the  camera.  To  avoid  dealing  with  a  reversed  image,  the  image  plane  is  often  considered  to  lie  in  front  of 
the  camera,  at  the  same  distance  from  the  pinhole.  The  distance  from  the  focal  point  to  the  image  plane 
is  the  focal  length. 

A  perfect  lens  can  be  modeled  as  a  pinhole.  No  lens  is  perfecL  of  course,  so  part  of  the  problem 
of  geometric  camera  calibration  is  correcting  for  lens  distortions.  The  most  accurate  and  conceptually 
simple  method  of  camera  calibration  would  be  to  measure  calibration  parameters  at  each  pixel  in  the 
image.  For  example,  at  each  pixel  measure  the  line-of-sight  vector.  This  would  produce  a  gigantic  lookup 
table.  Then,  given  a  pixel  in  an  image,  simple  indexing  would  yield  the  line-of-sight  vector  that  solves 
the  back-projection  problem.  To  solve  the  projection  problem,  the  table  would  be  searched  to  find  the 
line-of-sight  vector  that  passes  nearest  the  point  in  question. 

A  lookup  table  of  calibration  data  for  each  pixel  would  be  prohibitively  expensive.  The  obvious 
compromise  is  to  sample  the  image,  and  interpolate  between  data  points.  If  the  error  in  interpolation  is  less 
than  the  measurement  error,  no  accuracy  is  losL  Most  approaches  to  geometric  camera  calibration  involve 
sampling  the  image,  and  solving  for  the  parameters  of  the  interpolation  functions.  The  obvious  differences 
between  approaches  are  in  the  frm  of  the  interpolation  functions,  and  the  mathematical  techniques  used 
to  solve  for  the  parameters.  The  main  intent  of  most  calibration  work  has  been  the  solution  of  the  back- 
projection  problem.  The  projection  problem  has  occasionally  been  overlooked.  The  following  paragraphs 
briefly  describe  past  work. 
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Figure  1:  The  Pinhole  Camera  Model 


•  Sobel  [S]  introduced  a  method  for  calibration  that  involved  the  solution  of  a  large  system  of  non¬ 
linear  equations.  In  addition  to  solving  for  the  intrinsic  camera  parameters,  his  method  also  solved 
for  extrinsic  camera  parameters,  such  as  camera  pan  attd  tilt  Sobel  used  the  basic  pinhole  model, 
and  solved  the  system  using  a  non-linear  optimization  method.  He  did  not  model  lens  distortions, 
and  the  system  depended  on  the  user  to  provide  initial  parameters  for  the  optimization  technique. 

Tsai  [7]  improved  on  the  general  non-linear  approach  in  several  ways.  He  modeled  distortions 
globally  using  fourth  order  functions,  and  presented  a  method  for  computing  good  initial  parameters 
for  the  optimization  technique.  Tsai’s  model  of  lens  distortions  assumes  that  the  distortions  are 
radially  symmetric. 

•  Yakimovsky  and  Cunningham  [9]  presented  a  calibration  technique  that  also  used  a  pinhole  model  for 
the  camera.  They  treated  some  combinations  of  parameters  as  single  variables  in  order  to  formulate 
the  problem  as  a  system  of  linear  equations.  However,  in  this  fcmnulation,  the  variables  are  not 
completely  linearly  independent,  yet  are  treated  as  such.  No  lens  disttstions  are  modeled  with  this 
approach. 

•  Martins,  Birk,  and  Kelley  [3]  reported  a  calibration  technique  that  does  not  utilize  an  explicit  camera 
model.  Their  two-plane  calibration  method  consisted  of  measuring  the  calibration  data  for  various 
pixels  across  the  image.  The  data  for  other  pixels  is  computed  by  interpolatitHi.  The  back-projection 
problem  is  solved  by  computing  the  vector  that  passes  through  the  interpolated  points  on  each 
calibration  plane.  The  interpolation  can  be  either  local  or  global.  The  two-plane  method  solves  only 
the  back-projection  problem. 

Isaguirre,  Pu,  and  Summers  [2]  extended  the  two-plane  method  to  include  calibration  as  a  function 
of  the  position  and  orientation  of  the  camera.  They  used  an  iterative  approach  based  on  Kalman 
filten  to  obtain  the  solution. 

Our  goal  in  camera  calibration  was  to  develop  a  single,  basic  calibration  procedure  to  solve  both  the 
projection  and  back-projection  problems  for  a  variety  of  applications.  Consequently,  the  desired  procedure 
had  to  be  conceptually  straightforward,  easily  extended  to  obtain  various  degrees  of  accuracy,  and  compu¬ 
tationally  efficient.  To  meet  these  requirements,  we  chose  to  begin  with  the  two-plane  method  of  Martins, 


122 


Birk.  and  Kelley.  Section  2  discusses  the  two-plane  method  and  the  solution  to  the  back-projection  prob¬ 
lem.  This  method  can  be  made  arbitrarily  accurate:  the  only  problem  is  that  it  fails  to  solve  the  projection 
problem.  In  section  3  we  present  a  method  for  solving  the  projection  problem  that  utilizes  the  calibration 
data  from  the  two-plane  method.  The  solution  to  the  projection  problem  is  a  simple  application  of  analytic 
geometry,  and  is  completely  formulated  with  systems  of  linear  equations. 

The  calibration  method  presented  in  this  p£q)er  has  been  implemented  and  tested  in  the  Caliixated 
Imaging  Laboratory  at  CMU  [4].  Results  are  presented  in  section  4  that  demonstrate  the  accuracy  of  diis 
method. 
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Figure  2:  Two-Plane  Calibration 

2  The  Solution  to  the  Back>Projection  Problem 

Martins,  Birk,  and  Kelly  [3]  first  formally  presented  the  two-plane  calibration  technique  for  solving 
the  back-projection  problem.  This  technique  has  the  advantage  that  it  provides  exactly  the  information 
needed — the  ray  in  space  that  defines  the  line  of  sight  of  a  given  pixel — without  any  explicit  camera  model. 

Figure  2  illustrates  the  concept  of  two-plane  calibration.  Let  P\  and  Pi  denote  the  calibration  planes. 
Assume  that  the  3d  locations  of  the  calibration  points  on  each  plane  are  measured.  An  image  of  each 
plane  is  acquired,  and  the  image  location  of  each  of  the  calibration  points  is  extracted.  Let  the  calibration 
points  be  denoted  py,  and  the  corresponding  image  locations  be  denoted  9,y,  where  i  =  1, 2  is  the  plane, 
and  y  =  1, 2, . . . ,  n  is  the  point  index.  Thus,  the  image  of  py  is  qy. 

Let  p  and  7  denote  the  row  and  column  coordinates,  respectively,  of  an  image.  Then,  given  a  point 
V  =  [  P  7  ]*  in  the  image,  the  line-of-sight  vector  for  v  can  be  computed  as  follows.  First,  use  the 
points  p\j  and  qij  to  interpolate  the  location  of  v  on  the  first  calibration  plane.  Pi.  Call  this  point  ui. 
Then,  interpolate  to  find  the  location  of  v  on  the  second  calibration  plane.  Call  this  point  uj.  The  pixel 
line-of-sight  vector  then  has  direction  ui  -  ui  and  passes  through  the  point  ui . 

Various  types  of  interpolation  can  be  used,  with  different  degrees  of  accuracy.  Martins,  et  al  report 
three  types  of  int»polation:  linear,  quadratic,  and  linear  spline.  The  two-plane  method  has  the  potential 
for  being  tiie  most  accurate  of  any  calibration  method  for  the  solution  of  the  back-projection  problem.  At 
the  limit,  this  technique  consists  of  measuring  the  line-of-sight  vectors  for  each  pixel  in  the  image.  As  will 
be  seen  in  Section  4,  the  number  of  calibration  points  used  has  a  strong  influence  on  the  accuracy  of  the 
calibration. 

2.1  Global  Interpolation 

One  approach  to  interpolating  the  calibration  data  is  to  globally  fit  an  interpolation  function  to  the  data. 
This  function  is  then  used  for  any  pixel  across  the  entire  image.  Global  interpolation  has  the  effect  of 
averaging  errors  over  all  the  pixels  so  that  the  resultant  line-of-sight  vector  is  exact  for  no  pixel,  but  is  close 
for  all  pixels.  This  has  the  advantage  of  reducing  the  sensitivity  to  errors  or  noise  in  measurements.  On 
the  other  hand,  the  form  of  the  interpolation  function  is  an  a  priori  assumption  about  the  lens  distortions, 
and  may  or  may  not  be  appropriate. 
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2.1.1  Linear  Interpolation 

Let  pij  =  {  X  y  z  ]‘,  and  qij={  p  7  1  }'.  Then  a  linear  transformation  between  p  and  q  is  given  by 

Pij  ~  ^iqij 

where  4,  is  a  3x3  matrix.  Given  n  measurements  on  each  planet  we  can  then  form  the  system 

[  Pa  Pa  •••  Pin]  =4.[  qa  qa  ...  ?«  ] 


or, 

Pi  =  AiQi 

This  system  can  be  solved  in  the  least  squares  sense  by  using  the  matrix  pseudoinverse  (also  called  the 
generalized  matrix  inverse)  [1]: 

Ai=PiQ’[QiQ^\~' 

Given  a  pixel  v  in  the  image,  the  direction  of  the  line-of-sight  vector  through  v  is  given  by  u\  —  ut 
where  ui  =  Ax  v,  and  uj  -  Aav. 


2.1.2  Quadratic  Interpolation 

Quadratic  interpolation  is  similar  to  linear,  except  that  second-order  tenns  are  used  in  the  parameterization, 
and  the  matrix  A  is  6x6.  We  represent  a  point  in  space  hy  p  =  [  x  y  z  ]  ,  but  we  represent  image 
locations  by  ^  =  [  p  7  7^  P7  1  ]*.  With  these  modifications,  the  formulation  is  otherwise 

identical.  Martins,  et  al  report  that  quadratic  interpolation  was  more  accurate  than  linear. 

2.2  Local  Interpolation 

With  no  a  priori  knowledge  about  the  lens  distortions,  global  interpolation  may  be  inappropriate.  A  better 
approach  may  be  to  model  the  distortions  locally.  If  the  calibration  data  is  dense  enough,  the  interpolation 
can  be  very  accurate.  In  the  paragraphs  below,  we  discuss  a  technique  called  linear  spline  interpolation, 
which  uses  a  linear  function  to  perform  interpolation  over  each  local  region. 

Conceptually,  this  technique  of  interpolation  consists  of  tesselating  each  calibration  grid  with  triangles, 
and  performing  linear  interpolation  within  each  triangle.  The  calibration  points  form  the  vertices  of  the 
triangles.  A  plane  is  defined  uniquely  by  three  points,  so  no  errors  are  introduced  at  the  vertices.  This 
is  not  the  case  for  global  interpolation  techniques,  in  which  errors  are  avoaged  over  all  points,  including 
calibration  points.  Martins,  et  al  achieved  their  best  accuracy  using  this  form  of  interpolation.  In  section 
4,  we  report  experiments  which  confirm  this  result 

In  our  current  implementation,  the  grid  is  not  tesselated  in  advance.  Instead,  for  any  point  v  in  the 
image,  each  calibration  grid  is  searched  to  find  the  three  closest  calibration  points.  The  linear  interpolation 
matrices  A,  are  computed  using  just  three  points  each.  The  line-of-sight  vector  is  then  computed  as  in 
Section  2.1.1.  Figure  3  illustrates  the  procedure. 
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3  Linear  Solution  to  the  Projection  Problem 


The  two-plane  method  for  solution  to  the  back-projection  problem  did  not  utilize  an  explicit  camera  model. 
In  order  to  use  the  two-plane  data  to  obtain  a  solution  to  the  projection  problem,  it  is  necessary  to  have  a 
camera  model  to  formulate  the  equations.  We  use  a  model  similar  to  that  of  Yakimovsky  and  Cunningham 
[9]. 

In  figure  4  we  begin  with  a  pinhole  model  and  define  the  following  vectors  and  points: 

•  P  =  [  Px  Py  Pf  ]  *  =  the  vector  from  the  origin  to  a  point  ui  space. 

»  F  =[  fx  fy  /x  ]  ‘  =  the  vector  from  the  csigin  to  the  camera  focal  point 

•  p  =  [  Tx  ty  r,  ]  ‘  =  a  vector  that  points  along  the  direction  of  increasing  row  number.  R  represents 

the  displacement  vector  from  one  pixel  to  the  next  in  the  row  direction.  The  magnitude  of  P  is  the 
row  scale  factor. 

•  C  s  [  Cx  Cy  c,  ] '  =  a  vector  that  points  along  the  direction  of  increasing  column  number.  C 
represents  the  displacement  vector  from  one  pixel  to  the  next  in  the  column  direction.  The  magnitude 
of  C  is  the  colurim  scale  factor. 

•  [  Pp  7p  ]  —  die  piercing  point  of  the  image,  or  the  point  where  the  optical  axis  pierces  the  image 
plane. 

The  vectors  P  and  C  define  the  raientation  and  scale  of  the  image  plane.  Columns  in  the  image  plane 
are  parallel  to  P,  while  rows  are  parallel  to  C. 

The  projection,  [  p  7  ] ,  of  a  point  P  onto  the  image  plane  can  be  computed  by  taking  the  dot  product 
of  the  vector  from  the  focal  point  to  P,  and  adding  the  offset  to  the  piercing  point  Per  example,  consider 
computing  the  row  coordinate,  p,  of  the  projection.  The  vector  P  -  F  is  the  vector  from  the  focal  point 
that  passes  through  P.  Every  point  along  this  vector  will  have  the  same  location  in  the  image.  Let  V  be  a 
normalized  vector  along  P~  F,  that  is,  let 


V  = 


F-F 

\\p-n 


(1) 


where  ||  ||  denotes  the  length  of  a  vector.  Let  0  represent  the  usual  vector  dot  product  Then  V  0  P 
represents  the  projection  of  V  onto  R  measured  in  row  units.  Adding  the  row  coordinate  of  the  piercing 
point  translates  V  0  P  into  image  coordinates. 

Therefrae,  the  image  location,  [  p  7  ] ,  of  a  point  P  can  be  computed  using  the  equations: 


p  =  V'  0  P  +  Pp  (2) 

7  =  V0C  +  7^  (3) 

If  F,  P,  C,  Pp,  and  7^  are  all  unknown,  then  the  resulting  system  is  non-linear.  However,  the  two-plane 
formulation  of  the  back-projection  problem  yields  the  information  needed  to  make  solving  for  the  focal 
point  location  a  linear  problem. 


3.1  Focal  Point  Solution 


In  a  pinhole  camera,  all  incoming  light  rays  pass  through  the  focal  point.  Since  a  lens  is  not  a  perfect 
pinhole,  we  instead  refer  to  an  effective  focal  point,  which  is  the  point  that  is  closest  to  all  the  rays.  From 
the  two-plane  method  for  the  back-projection  problem,  one  can  compute  a  bundle  of  rays  that  pass  through 
the  lens.  The  next  step  is  to  find  the  point  in  space  that  minimizes  the  distance  to  aU  the  rays. 

The  equation  for  the  squared  distance,  tfi,  from  a  point  F  =  [  x  y  z  ] '  to  the  line  through  Pi  = 
[  xi  yi  zi  ]' in  direction  [  a  b  c  ]' (where  =  1)  is: 


y-yi 

b 


z  —  Zt 
c 


Z-Zi 

c 


X-  XI 

a 


X-  Xi 

a 


y-yi 

b 
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Figure  4:  A  Linear  Model  of  Camera  Geometry 


Expansion  of  terms  yields: 

S  -  +  c^)  +  +  c^)  +  z^(a^  +  6^) 

-  Ix^ab  -  hczac  -  lyzbc 
+  2x(bki  -  cki)  +  2y(ck\  -  aki)  +  2z(qki  -  bk\) 


where: 


k\  =  zifr-yic 
kz  =  xic-zia 
k^  =  y\a  -  xib 

To  find  the  effective  focal  point,  we  need  to  minimize  D  =  Y^c^.  Differentiating  D  with  respect  to  x, 
y,  and  z  yields: 

dD/dx  =  ^  2x(b^  +  ^  2yab  —  ^  2zac  +  ^  2{bkz  —  ckz) 

dD/dy  =  ^  2y(a^  +  c^)  -  ^  2xab  —  ^  2zbc  +  ^  2(cifci  -  ak^) 
dDidz  =  ^  2z(a^ +  h^)  — ^  2iac— ^  2>'6c  +  ^  2(0^2  -  Wi) 


The  sums  are  taken  over  all  the  line-of-sight  vectors  (a,  b,  c,  k\,  kz,  kz  are  functions  of  the  vectors). 

Now,  by  setting  the  derivatives  of  D  to  zero  to  find  the  minima,  and  puLng  the  equations  in  matrix 
form,  we  obtain: 

h  =  Af 


where: 


h=> 


"Zickz  -  bki) 
Y^iaki  -  ck\) 
Yibkx  -  akz) 
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A  = 


+  c^)  -^ab 
-Ylab  +  c^)  -  E 

-Hoc  -E*c  E(a^  +  ^) 


/  = 


So  the  solution  we  seek./,  the  effective  focal  point  of  the  camera,  is  simply: 

f-A-^h 

3.2  Computation  of  the  Camera  Base  Vectors 

Equations  (2)  and  (3)  relate  the  position  of  a  point  in  space  to  a  corresponding  image  location.  These 
equations  can  be  written  as  a  linear  system: 


[  P  7  ]  =  [  V,  Vy  V,  1.0  ] 

Given  N  points  in  space,  we  have  2N  equations  to  solve  for  the  8  unknowns  in  R,  C,  pp,  and  7p: 


r,  c, 

Ty  Cy 
Fi  C, 

VPp  Tp  J 


5 

’  V,1  Vyl  V,1  1.0  ■ 

r,  t^x 

Pi  72 

va  v,2  1.0 

r,  Cy 

j  ; 

:  :  :  ; 

r,  c. 

.  p»  "y" . 

.  V„  Vy,  v„  1.0  . 

or, 

B-WX 

So,  using  the  pseudoinverse  to  obtain  a  least  squares  solution,  we  have: 

=  [jy'lF]”'  WB 

X  contains  the  values  for  R,  C,  pp,  and  7p. 


3  J  The  Local  Projection  Problem 

In  section  2  we  presented  several  ways  of  modeling  camera  gemnetry  for  the  back-projection  problem. 
The  linear  interpolation  technique  (section  2.1.1),  which  involved  fitting  a  first-order  transformation  to  all 
the  calilvation  data,  is  a  global  modeling  technique.  The  linear  spline  interpolation  technique  (section  2.2) 
is  a  local  modeling  technique,  since  at  each  pixel,  only  the  calibration  data  in  a  local  regicm  around  the 
pixel  is  used  to  compute  the  interpolation  function.  The  results  of  Martins,  et  al  [3],  and  our  laboratory 
results  (see  section  4)  both  indicate  that  a  local  modeling  technique  yields  superior  accuracy. 

The  solution  to  the  projection  problem  presented  in  sections  3.1  and  3.2  is  a  global  modeling  technique. 
All  the  calibration  data  is  used  to  compute  the  model  parameters,  and  the  results  are  used  to  solve  the 
projection  problem  for  any  point  in  space.  In  direct  analogy  to  the  linear  spline  technique  used  in  back- 
projection,  a  technique  can  be  derived  for  using  local  information  to  improve  the  accuracy  of  solution  to 
the  projection  problem. 

Our  local  projection  technique  involves  finding  a  linear  model  for  local  regitms  of  the  image.  The 
technique  involves  two  steps.  In  the  first  step,  the  global  solution  is  used  to  obtain  an  estimated  image 
location  for  a  point  in  space.  That  estimated  image  location  is  used  to  find  the  four  nearest  calibration 
points  on  each  calibration  plane.  These  points  are  used  to  compute  a  local  linear  solution  to  the  projection 
problem.  The  local  linear  solutions  could  also  be  [recomputed,  and  the  global  solution  would  then  be  used 
simply  to  index  the  correct  local  solution. 
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4  Experimental  Results 

Measurements  and  tests  were  conducted  within  the  Calibrated  Imaging  Laboratory  (CIL)  at  CMU  (Shafer 
[4]).  The  CIL  is  a  facility  that  provides  a  precision  imaging  capability.  The  purpose  of  the  CIL  is  to 
provide  researchers  with  accurate  knowledge  about  ground  truth  so  that  computer  vision  theories  can  be 
tested  under  controlled  scientific  conditions.  Of  particular  interest  for  this  study,  the  CEL  provides  facilities 
to  accurately  measure  point  locations,  and  to  accurately  position  and  orient  cameras. 

Position  measurement  of  points  in  the  CIL  is  performed  with  the  use  of  theodolites  (surveyor’s  transits), 
which  are  basicaUy  telescopes  with  crosshairs  for  sighting,  mounted  on  accurate  panAilt  mechanisms. 
Objects  to  be  measured  are  placed  at  one  end  of  an  optical  bench;  the  th^olites  are  fixed  to  the  other 
end,  separated  by  a  little  more  than  1  meter.  To  measure  the  position  of  a  point,  the  crosshairs  of  each 
theodolite  are  placed  over  the  point,  and  the  horizontal  and  vertical  displacements  read  off.  Trigonometric 
equations  then  yield  the  position  of  the  point  in  a  Cartesian  coordinate  system  defined  with  respect  to  the 
theodolites.  As  currently  configured,  the  theodolites  can  determine  point  locations  to  less  than  0.1  mm. 

4.1  Test  Scenario 

The  laboratory  tests  described  below  were  designed  to  provide  answers  to  the  following  questions: 

1.  What  accuracies  can  be  expected  from  off  the  shelf  cameras  and  lenses? 

2.  How  does  increasing  the  number  of  calibration  points  affect  the  accuracy  of  calibration? 

3.  What  is  the  expected  accuracy  for  the  projection  problem? 

Tests  were  performed  using  a  calibrated  grid.  The  grid  consisted  of  horizontal  and  vertical  lines  1mm 
in  width,  spaced  12.7  mm  apart  The  intersections  of  the  lines  on  the  grid  were  used  as  calibration  points. 
A  special  intersection  detector  was  implemented  to  extract  the  intersections  from  digital  images  with  sub¬ 
pixel  precision.  Each  time  the  grid  was  moved,  new  measurements  were  taken,  an  image  digitized,  and 
the  intersection  detector  applied.  The  result  was  a  data  file  in  which  each  calibration  point  was  associated 
with  its  3d  position  and  its  image  location. 

A  complete  test  consisted  of  data  from  three  different  grid  locations.  Due  to  the  size  of  the  laboratory, 
focal  length  of  the  lens,  and  depth  of  field  of  the  lens,  the  grid  was  typically  placed  at  distances  ranging 
from  O.SOm  to  0.56m  from  the  camera.  Data  from  two  of  the  grid  locations  was  used  to  compute  calibration 
parameters.  These  parameters  were  then  tested  using  data  from  the  third  grid  location.  The  third  grid  will 
often  be  referred  to  as  the  test  grid.  In  each  of  the  tests  reported  here,  the  focus  of  the  camera  was  kept 
fixed.  The  camera  used  was  a  Sony  CCD,  model  AVC-Dl,  with  the  standard  I6mm  lens. 

A  total  of  300  calibration  points  were  used  on  each  grid.  Rather  than  measure  the  location  of  each 
point  individually,  the  location  of  each  point  was  computed  based  on  the  measured  locations  of  the  center 
point  and  the  four  comers.  Consequently,  the  accuracy  of  the  data  depended  not  only  on  the  accuracy  of 
the  theodolites,  but  also  upon  factors  such  as  the  planarity  of  the  grid,  and  the  precision  of  the  grid  lines. 
In  preparing  for  each  test,  the  overall  accuracy  of  the  calibration  data  was  estimated.  For  several  points 
at  each  grid  location,  the  3d  locations  were  measured  using  the  theodolites.  The  measured  locations  were 
then  compared  with  the  computed  locations.  Differences  of  up  to  0.2  mm  were  recorded,  with  typical 
differences  being  between  0.1  and  0.2  mm.  The  accuracy  of  the  calibration  method  is  limited  by  the 
accuracy  of  the  calibration  data,  so  the  best  accuracy  achievable  in  this  scenario  is  between  O.I  and  0.2 
mm. 

The  effects  of  density  of  calibration  points  on  calibration  accuracy  was  tested  by  varying  the  number 
of  calibration  points  used.  This  was  easily  implemented  by  simply  skipping  over  some  of  the  rows  and 
columns  in  the  grid.  In  each  case,  the  calibration  points  were  uniformly  disuibuted  over  the  image.  Data 
is  reported  for  the  following  distributions  of  points:  3x3,  5x7, 7x10,  15x20. 

In  all  the  tests  reported  below,  grid  0  refers  to  the  grid  location  farthest  from  the  camera,  while  grid  2 
refers  to  the  grid  location  closest  to  the  camera.  In  all  cases,  the  number  of  points  was  varied  to  compute 
the  calibration  parameters,  but  all  300  calibration  points  on  the  test  grid  were  used  in  testing. 
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array 

calibration 

test 

error  (mm) 

size 

grids 

grid 

global 

local 

3x3 

0,  1 

2 

1.921 

0.731 

0.2 

1 

0.388 

0.296 

1.2 

0 

0.561 

0.317 

5x7 

0.  1 

2 

1.854 

0.740 

0.2 

1 

0.366 

0.166 

1.2 

0 

0.551 

0.235 

7x10 

0.  1 

2 

1.810 

0.696 

0.2 

1 

0.350 

0.169 

1.2 

0 

0.509 

0.185 

15x20 

0.  1 

2 

1.813 

0.666 

0.2 

1 

0.366 

0.147 

1.2 

0 

0.534 

0.201 

Table  1:  Calibration  Accuracy  of  the  Back-Projection  Problem 


4.2  Back-Projection  Results 

To  test  the  accuracy  of  the  back-projection  problem,  the  image  location  of  each  of  the  calibration  points 
on  the  third  grid  was  used  to  compute  a  line-of-sight  vector.  The  intersection  of  this  vector  with  the  plane 
of  the  test  grid  was  computed,  and  the  distance  between  the  intersection  and  the  actual  position  was  used 
as  the  error  measure.  In  the  results  reported  below,  the  errors  are  averages  taken  over  all  the  calibration 
points. 

Table  1  presents  the  results  obtained  for  the  back-projection  problem.  The  first  error  column  contains 
the  results  for  global  linear  interpolation.  For  this  method,  the  density  of  the  calibration  grid  makes  little 
or  no  differetKe  to  the  accuracy  of!  the  result  This  was  expected;  since  the  calibration  points  are  uniformly 
distributed  across  the  grid,  additional  points  do  not  provide  additional  information  for  a  linear  fit  The  best 
accuracy  is  achieved  when  the  test  grid  is  positioned  between  the  other  two  grids  used  for  calibration. 

The  second  error  column  in  table  1  presents  the  results  obtained  for  back-projection  problem  using  local 
linear  spline  interpolation.  This  time  there  is  a  general  trend  for  greater  accuracy  with  more  calibration 
points.  This  reflects  the  fact  that  the  linear  spline  method  interpolates  over  local  regions,  and  can  more 
accurately  approximate  effects  such  as  barrel  distortion.  There  are  instances  observable  in  the  table  which 
seem  to  contradict  the  general  trend;  these  are  most  likely  due  to  noise  in  the  measurements  or  in  the 
process  of  point  extraction.  Over  a  number  of  trials,  the  general  trend  has  been  consistent 

The  results  in  table  1  agree  with  the  results  obtained  by  Martins,  et  al.  To  summarize,  the  local  linear 
spline  interpolation  procedure,  with  as  few  as  12  calibration  points,  is  more  accurate  than  global  linear 
interpolation.  In  addition,  the  use  of  more  calibraticm  points  improves  the  accuracy  of  the  local  linear  spline 
method.  The  accuracies  we  achieved  in  our  tests  were  at  the  level  of  the  accuracies  of  our  measurements. 

4.2.1  Projection  Results 

The  accuracy  of  the  projection  problem  was  tested  with  a  procedure  similar  to  that  used  in  the  back- 
projection  problem.  Tlie  3d  location  of  each  calibration  point  on  the  test  grid  was  projected  into  the  image 
phots,  and  the  difference  (in  pixels)  between  the  project^  location  and  the  measured  location  was  used  as 
the  error  measure. 

The  test  results  for  the  projection  problem  are  reported  below  in  table  2.  The  fint  error  column  gives 
the  error,  in  pixels,  of  the  accuracy  using  global  interpolation.  The  average  error  reported  in  all  cases  was 
less  than  two  pixels,  which  is  good  enough  for  many  applications.  The  results  indicate  that  the  standard 
lenses  for  our  cameras  are  reasonably  good,  and  can  be  approximated  well  with  a  pinhole  model. 

The  second  error  column  in  table  2  reports  the  errors  recorded  using  the  local  solution  to  the  projection 
problem. 
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array 

calibration 

test 

error  (pixels) 

size 

grids 

grid 

global 

local 

3x3 

2 

1.58 

1.70* 

1 

0.65 

warn 

0 

0.89 

0.82 

5x7 

2 

1.32 

1 

0.95 

mSM 

0 

0.98 

7x10 

2 

■rohI 

1 

0 

15x20 

■ESIi 

2 

■RM 

1.38* 

1 

■al 

0 

Table  2:  Accuracy  of  the  Projection  Problem 


A  comparison  of  the  two  columns  in  table  2  shows  an  improvement  resulting  from  using  local  infor¬ 
mation.  In  general,  the  results  from  the  local  solution  to  the  projection  problem  are  a  factor  of  2  improved 
over  the  global  solution.  The  entries  followed  by  a  *  are  examples  where  the  global  result  was  better  than 
local  result;  this  may  be  an  effect  of  errors  in  the  measurement  process.  The  general  conclusion  that  can 
drawn  is  that  local  models  of  camera  geometry  provide  mote  accurate  results  than  global  models — ^for 
iple  interpolation  functions. 

4.2,2  Conclusions 

In  section  4.1,  we  enumerated  three  questions  which  were  to  be  answered  by  the  tests  reported  above.  We 
now  proceed  to  answer  each  of  these  questions  in  turn. 

1.  What  accuracies  can  be  expected  from  off  the  shelf  cameras  and  lenses? 

Tables  1  and  2  of  test  results  show  the  accuracy  achievable  with  a  standard  conunercial  CCD,  using 
the  standard  lens  supplied  with  the  camera.  With  a  simple  global  interpolation  scheme,  accuracies 
as  good  as  1  part  in  1400  (0.3  mm  over  530  mm)  can  be  obtained.  With  a  more  sophisticated  local 
linear  spline  interpolation,  the  accuracies  can  be  increased  to  1  part  in  3500. 

2.  How  does  increasing  the  number  of  calibration  points  effect  the  accuracy  of  calibration? 

We  have  shown  that  the  maximam  accuracy  fex’  global  linear  interpolation  can  be  achieved  with  a 
small  number  of  calibration  points,  provided  that  the  points  are  uniformly  distributed  over  the  image. 
Further  increasing  the  number  of  calibration  points  has  no  effect  on  the  accuracy.  With  a  local  linear 
spline  interpolation,  adding  calilnation  points  clearly  improves  the  accuracy  of  the  back-projection 
problem,  until  the  limiting  accuracy  of  the  calibration  data  is  reached. 

3.  What  is  the  expected  accuracy  for  the  projection  problem? 

Using  either  a  local  or  global  solution,  the  projection  problem  can  be  solved  to  within  two  pixels; 
results  as  good  as  0.34  pixels  were  report^  For  many  applications,  solution  of  the  projection 
problem  need  not  be  extremely  accurate.  In  many  instances,  the  projected  pixel  location  is  only 
needed  to  find  the  center  of  a  region  within  which  an  operation  will  be  p^ormed.  For  these 
applications,  accuracy  of  one  to  two  pixels  is  adequate. 

It  is  important  to  note  that  the  local  interpolaticm  outpoformed  the  global  interpolation.  While  the 
differences  were  not  great  in  our  tests,  the  lenses  we  used  were  fairly  linear.  If  extremely  wide  angle 
lenses  are  used,  the  distortions  may  be  large,  and  the  ability  to  locally  interpolate  will  be  much  more 
important. 
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5  Discussion 

We  have  presented  a  calibration  method  that  we  believe  meets  many  of  the  requirements  of  a  basic 
calibration  technique  that  can  be  used  for  a  variety  of  applications.  Our  method  is  based  on  the  two-plane 
method  of  Martins,  Birk,  and  Kelley  [3],  but  is  ext«ided  to  include  a  solution  to  the  projection  problem. 
We  believe  that  the  method  presented  here  has  many  advantages,  described  in  the  following  paragraphs; 

•  Completeness. 

The  oiginal  two-plane  method  of  calibration  only  provided  a  solution  to  tiie  back-projection  problem. 
While  this  is  sufficient  for  many  applications,  a  solution  to  the  projection  problem  is  also  necessary 
for  applications  such  as  mobile  robots.  We  have  extended  the  two-plane  method  by  providing  a 
solution  to  the  projection  problem. 

•  Accuracy 

The  two-plane  calibration  method  can  be  made  arbitrarily  accurate.  As  reported  in  section  4,  in¬ 
creasing  the  number  of  calibration  points  results  in  increasing  accuracy.  If  no  improvement  results 
from  adding  mote  points,  then  the  accuracy  of  the  calibration  data  must  be  improved. 

The  projection  problem  exhibits  much  of  the  same  behavior  as  the  back-projection  problem.  Of 
particular  interest  is  the  observation  that  local  modeling  of  camera  geometry  improves  the  accuracy 
of  the  projection  problem,  as  well  as  the  back-projection  problem.  The  accuracies  observed  in  our 
tests  were  typically  less  than  one  pixel. 

•  Simplicity. 

The  two-plane  model  is  conceptually  v^  straightforward  and  easy  to  implement  The  use  of  the 
line-of-sight  vectors  to  solve  for  the  parameters  of  a  linear  camera  model  arises  intuitively  fiom  the 
geometry  of  the  model.  The  method  of  solution  for  the  camera  model  involves  solving  only  linear 
equations,  so  no  sophisticated  optimization  techniques  are  involved. 

•  Efficiency. 

Solution  of  either  the  back-projection  or  projection  problems  require  only  a  few  matrix  multiplies 
and  matrix  inversions  on  small  matrices.  The  operations  are  guaranteed  to  produce  a  unique  answer 
within  a  fixed  time.  While  a  relatively  large  amount  of  data  must  be  stored  for  this  calibration 
method  compared  to  other  methods,  the  total  amount  is  still  insignificant 

•  Practicality. 

Because  the  method  provides  a  complete  solution  to  the  geometric  calibration  problem,  the  method 
can  be  used  for  any  application.  The  accuracy  can  be  arbitrarily  increased  (or  decreased)  to  meet 
the  requirements  for  a  given  application.  The  oidy  change  in  the  method  is  to  store  the  data  finom 
more  calibration  points.  The  mathematics  remains  the  same,  and  no  special  equipment  is  required 
beyond  that  needed  to  obtain  precise  locations  for  the  calibration  points. 

In  addition  to  the  benefits  of  the  method  we  presented,  some  general  observations  should  be  made: 

•  Without  the  benefit  of  a  priori  knowledge  of  the  form  of  lens  distortions,  local  modeling  of  distortions 
seems  to  perform  better  than  global  modeling.  A  global  model  is  an  attempt  to  fit  the  data  into  a 
predetermined  form  and  average  the  error  across  the  entire  image.  The  accuracy  of  the  interpolation 
is  limited  by  how  well  the  chosen  model  reflects  reality.  Conceivably,  a  different  function  could  be 
required  for  different  types  of  lenses  to  reflect  different  models  of  distortion. 

Modeling  distortion  locally  makes  no  assumption  about  the  forms  of  the  lens  distortions.  The  local 
model  can  be  made  arbitrarily  accurate  by  simply  sampling  at  more  pixels.  We  have  shown  that 
relatively  few  points  are  needed  to  achieve  the  level  of  accuracy  that  the  measurement  devices 
provide.  Moreover,  local  modeling  is  more  accurate  for  solving  both  the  projection  and  back- 
projection  problems. 
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•  Nearly  all  of  the  data  reported  showed  that  the  best  accuracy  was  obtained  when  the  test  grid  was 
placed  between  the  two  grids  used  for  calibration.  This  is  a  specific  instance  of  the  general  fact  that 
interpolation  is  more  accurate  than  extrapolation.  In  calibrating  a  real  robotic  system,  the  calibration 
data  should  ideally  be  obtained  so  as  to  bound  the  region  of  interest  as  much  as  possible. 

Geometric  camera  calibration  may  depend  on  a  variety  of  factors.  For  example,  the  focal  distance,  the 
aperture  setting,  presence  or  absence  of  a  filter,  or  even  the  operating  temperature  of  a  camera  may  all 
affect  the  calibration  parameters.  We  are  currently  making  measurements  and  conducting  tests  to  determine 
the  sensitivity  of  calibration  parameters  to  many  of  these  factors. 

The  results  reported  in  this  paper  were  obtained  using  data  from  the  Calibrated  Imaging  Laboratory  at 
CMU.  Our  next  application  of  this  method  will  be  to  calibrate  and  register  three  cameras  and  a  laser  range 
findo-  mounted  on  the  CMU  Navlab. 
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